JPL Publication 90-12 




i c4 i 
J & 

f 


An Application of Multiattribute 
Decision Analysis to the 
Space Station Freedom Program 

Case Study: Automation and Robotics 
Technology Evaluation 

Jeffrey H. Smith 
Richard R. Levin 
Elisabeth J. Carpenter 


May 1, 1990 


NASA 

National Aeronautics and 
Space Administration 

Jet Propulsion Laboratory 

California Institute of Technology 
Pasadena, California 

(NA^A-CR-l S74S2 ) AN APPLICATION OF 
MULTIATTRIBUTE DECISION ANALYSIS TO THE 
SPACE STATION FREEDOM PROGRAM. CASt STUDY: 
AUTOMATION AND RO n OTICS TECHNOLOGY 
EVALUATION (JPL) 196 p 


N9I-13A70 


Unci as 
0H7 t>4o 


CSCL 22B G3/18 





TECHNICAL REPORT STANDARD TITLE PAGE 


1 . Report No • 


90-12 


2. Government Accession No, 


4* Title and Subtitle 

An Application of Multiattribute Decision Analysis to 
the Space Station Freedom Program, .. Case Study: 
Automation and Robotics Technology Evaluation 


7 Author (s) I P® r f° rniin 9 Organization Report No, 

JeffreyH. Smith, Richard R. Levin, Elisabeth J. Carpenter 


3. Recipient's Catalog No, 


5. Report Date 

May 1. 1990 


6, Performing Organization Code 


9. Performing Organization Name and Address 

JET PROPULSION LABORATORY 
California Institute of Technology 
4800 Oak Grove Drive 
Pasadena, California 91109 


10, Work Unit No, 


11, Contract or Grant No. 

NAS7-9 18 


13. Type of Report and Period Covered 


12, Sponsoring Agency Name and Address 


NATIONAL AERONAUTICS AND SPACE ADMINISTRATION 
Washington, D.C. 20546 


14, Sponsoring Agency Code 
BH 476-30-70-08-00 


15, Supplementary Notes 


Ate 


16. Abstract X 

This repoTt describes the results of an application of multiattribute analysis to th^ 
evaluation of high-leverage prototyping technologies in the automation and robotics 
(A&R) areas that might contribute to the Space Station (S.S.) Freedom baseline design 

An implication of the study is that high-leverage prototyping is beneficial to the 
S.S. Freedom Program as a means for transferring technology from the advanced developmerj 
program to the baseline program. The. process also highlights the trade-offs to be made 
between subsidizing higlv^value, low- =: risk technology developments versus high-^value, higtf- 
risk technology developments. 

Twenty-one A&R Technology tasks spanning a diverse array of technical concepts were 
evaluated using multiattribute decision analysis. Because of large uncertainties 
associated with characterizing the technologies, the methodology was modified to Incor- 
porate uncertainty. Eight attributes affected the rankings: initial cost impacts, 
operations cost impacts, crew productivity, safety, resource requirements, growth poten-| 
tial, and spinoff potential. The four attributes of initial cost Impact, operations 
cost impact, crew productivity? and safety affected the rankings the most. 


Nine individuals, knowledgeable in Space Station Freedom technologies were interviewed 
to obtain their preferences. 

A questionnaire was sent to the participants to evaluate the process. While the 
participants had reservations about the manner in which the evaluation was transformed 
from an informal call for idease^ to a full*-scale technical evaluation, comments on the 
usefulness and appropriateness of the methods were positive. Future assessments should 


consider thp develonment and u 

se of decislo 

n siinnart f-nnla ti 

i fhp.i 

process — 

17. Key Words (Selected by Author(s)) 

Spacecraft Design, Testing, and Performance 
Behavioral and Social Sciences (General) ; 
Operations Research; Statistics and Proba- 
bility; Systems Analysis; R&D Planning; 
Space Station Freedom 

18* Distribution Statement 


19. Security Cl ass if, (of this report) 

20* Security Closslf. (of this page) 

21 . No. of Pages 

22. Price 

Unclassified 

Unclassified 

202 

JPL 0184 n 9/83 




JPL Publication 90-12 


An Application of Multiattribute 
Decision Analysis to the 
Space Station Freedom Program 

Case Study: Automation and Robotics 
Technology Evaluation 


Jeffrey H. Smith 
Richard R. Levin 
Elisabeth J. Carpenter 


May 1, 1990 


IXJASA 

National Aeronautics and 
Space Administration 

Jet Propulsion Laboratory 

California Institute of Technology 
Pasadena, California 


The research described in this publication was carried out by the Jet Propulsion 
Laboratory, California Institute of Technology, under a contract with the National 
Aeronautics and Space Administration. 

Reference herein to any specific commercial product, process, or service by trade name, 
trademark, manufacturer, or otherwise, does not constitute or imply its endorsement by 
the United States Government or the Jet Propulsion Laboratory, California Institute of 
Technology. 


ABSTRACT 


This report describes the results of an application of multiattribute decision analysis to the 
evaluation of high-leverage prototyping technologies in the automation and robotics (AfcR) areas that 
might contribute to the Space Station (S.S.) Freedom baseline design. 

An implication of the study is that high-leverage prototyping is beneficial to the S.S. Freedom. 
Program as a means for transferring technology from the advanced development program to the 
baseline program. The process also highlights the trade-offs to be made between subsidizing high- 
value, low-risk technology developments versus high-value, high-risk technology developments. 

Twenty-one AfcR technology tasks spanning a diverse array of technical concepts were evaluated 
using multiattribute decision analysis. Because of large uncertainties associated with characterizing the 
technologies, the methodology was modified to incorporate uncertainty. Eight attributes affected the 
rankings: initial cost impacts, operations cost impacts, crew productivity, safety, resource 

requirements, growth potential, and spinoff potential. The four attributes of initial cost impact, 
operations cost impact, crew productivity, and safety affected the rankings the most. 

Nine individuals, knowledgeable in Space Station Freedom technologies, were successfully 
interviewed to obtain their preferences. Rankings were calculated for the three groups they represented 
using forty-eight combinations of assumptions. The three groups were designated Level I, Level II, and 
third party A&R technology developers. 

Analysis showed that the ranking results were consistent across groups and for different model 
assumptions. The results were also in general agreement with separate evaluations conducted by the 
three working groups, although a number of the preferred alternatives of the working groups were rated 
at or near the bottom of the study list. 

A questionnaire was sent to the participants to evaluate the process. While the participants had 
reservations about the manner in which the evaluation was transformed from an informal call for ideas 
to a full-scale technical evaluation, comments on the usefulness and appropriateness of the methods 
were positive. Future assessments should consider the development and use of decision support tools 
to facilitate the process. 
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FOREWORD 


The Automation and Robotics Planning Task was established to provide support for analysis of 
Space Station Freedom automation and robotics (AfcR) issues with impacts on baseline 
implementation. The objectives of the task were to identify a process for A&R implementation, 
develop an A&R implementation plan, and initiate implementation of the process. 

This report describes a methodology for AfcR implementation with an application to a specific 
subset of A&R technologies-high-leverage prototypes. The term “high-leverage” refers to the ability of 
these technologies to produce large benefits for relatively small investments in research and 
development funds. The purpose of leveraging is to accelerate the technical readiness of the AfcR 
technologies for inclusion in the S.S. Freeedom design. 
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Introduction and Summary 


SECTION I 


INTRODUCTION 

AND 

SUMMARY 


A. PURPOSE 

This publication represents work in support of a Space Station (S.S.) Freedom Program process 
requirement for evaluation and implementation of automation and robotics (AfcR) technologies 
(Reference 1). During 1988, the S.S. Freedom Program undertook the development of an AfcR 
implementation plan to provide guidance for assuring that advanced A&R technologies were identified 
and incorporated into the program. A key element of the plan is a process for evaluating and choosing 
A&R candidates. 

A combination of external factors, program resource constraints, complex management structure, 
and technological uncertainties results in a complicated design environment for S.S. Freedom designers 
in the selection of A&R candidates. A formalized decision process for A&R candidates can help deal 
with much of the uncertainty and complexity, while maintaining focus on the ultimate objectives of 
incorporating A&R into the program. This decision process involves combining A&R technology data 
with value data (priorities) from S.S. Freedom Program management to obtain a ranking of the A&R 
candidates. The process described herein is based on a technique known as multiattribute decision 
analysis that has been applied in a number of research and development (R&D) programs to help guide 
technology research and decision making. This process was modified for application to, and possible 
adoption by, the S.S. Freedom A&R program. 

1. Objectives 

The purpose of this study is to develop an operational methodology for evaluation of A&R 
candidates that fulfills a number of objectives (Reference 2). The objectives of the study are to develop 
a method for the S.S. Freedom Program that (1) provides a consistent basis for evaluation; (2) 
provides a mechanism for reaffirming or checking the analyses of A&R candidate advocates; (3) 
provides an aggregation of inputs from all concerned parties, based on a consistent set of program 
evaluation attributes; and (4) provides the program director with a basis for making funding decisions. 

An implicit requirement of a method for analyzing AfcR candidates is the need to accommodate 
and quantify uncertainty. A common characteristic of S.S. Freedom A&R technology concepts is the 
lack of precedents. For many of the candidates, there is little, if any, historical data on which to 
evaluate cost and performance impacts and base estimates of future cost and performance. 
Consequently, the alternative AfcR candidates and their associated data must be represented with 
probabilistic estimates of their cost and performance. A subobjective of this study was to modify the 
multiattribute decision analysis methodology to accommodate uncertain (probabilistic) attributes. 
Technical management requires not only information about AfcR candidate value, but also information 
about the relative uncertainties that stem from incomplete knowledge of the technology and the 
likelihood that it will perform as required if funded for development. The modifications helped provide 
a window on the relationship between the mixtures of uncertainty associated with the different A&R 
candidates and how these uncertainties translate into the final measures of A&R candidate value. 
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2. Scope 

The scope of this study is applied to the evaluation of A&R candidates. While the same 
process can be used for evaluating non-A&R subsystem elements, the initial application described 
focuses on A&R because of the complex nature of AfcR systems integration. A&R technologies involve 
interfaces between multiple subsystems and, as such, face a number of implementation barriers. For 
example, different organizations and contractors are often responsible for different subsystems. Thus, 
inclusion of A&R elements requires formalized communication between different organizations and 
mediation of issues by a third party— typically the systems engineering function. In some ways, the 
study described in this publication is a demonstration of technical feasibility that A&R candidates of 
high potential value to the S.S. Freedom can be identified in a rigorous and credible manner. 

The study was applied to a specific class of A&R candidates— high-leverage prototype AfcR 
candidates at a fixed point in time. High-leverage prototypes are technologies with potential to provide 
substantial S.S. Freedom Program benefits for relatively small investments in research and 
development funds. The invested R&D funding is intended to accelerate the technical development of 
promising technologies for inclusion in the baseline design. It should be noted that the aim of this 
publication is to describe the evaluation process , the application of multiattribute decision analysis, 
and illustrate the process with an application. While the study touches upon a number of technical 
observations regarding the A&;R candidates, it does not focus on in-depth technical descriptions. Such 
descriptions are contained in an unpublished document used as input to the technical assessment 
process described in Section IV-E. Although the descriptions are not included in this report for 
proprietary reasons, the assessments of the technical assessors indirectly reflect these descriptions. 

3. Safeguards 

A common concern of participants in evaluation studies is the ability of the process to 
withstand artificial manipulation of inputs (data) in order to influence or bias the results to some 
advantage of a particular proponent (also known as “gaming” the system). This study was no 
exception. Although safeguards are an implicit objective of any evaluation, the level of concern 
expressed by some participants indicated that an additional objective was required of the methodology 
to minimize, by careful monitoring, any opportunities for gaming the process. The safeguards 
followed are further described in Section IV. 

B. BACKGROUND 

Background information is necessary to place the evaluation study in context and to present some 
of the unique factors that complicate the S.S. Freedom Program A&R implementation process. The 
United States Congress has directed the S.S. Freedom Program to identify and develop advanced 
automation and robotics technologies; to provide a Flight Telerobotic Servicer (FTS); and to report on 
these activities to Congress. The S.S. Freedom Program has specified program requirements for the 
development and accommodation of baseline A&R, and for the provision of hooks and scars for the 
addition of new candidates, in the Program Requirements Document (PRD) (Reference 1). In 
addition, the PRD specifically states that baseline applications of A&R (e.g., assignment of specific 
tasks to the FTS), as well as future applications, are to be based on program-developed criteria that 
address, at a minimum, the safety, productivity, and life-cycle cost benefits that are to be provided. 
The PRD also requires the program to develop an A&R implementation plan to address the process of 
A&R implementation in the program, 

“Baselined A&R” in the U.S. S.S. Freedom Program currently includes (1) provision of the FTS 
for assembly, maintenance, and servicing tasks, and (2) knowledge-based fault detection, isolation, and 
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recovery (FDIE) for the Operations Management System (OMS), and for general systems monitoring, 
diagnosis, redundancy management, test and checkout. In addition to the NASA elements, the 
international partners have committed to providing the following robotic devices: the Canadian S.S. 
Remote Manipulator System (SSRMS); the Canadian Special Purpose Dextrous Manipulator (SPDM); 
the remote manipulator system of the Japanese Experiment Module (JEM-RMS); and the Japanese 
small fine arm. 

The A&R Implementation Plan, currently under development by the S.S. Freedom Program 
Level II, defines the processes, activities, products, and responsibilities for meeting the program A&R 
requirements for baselined and candidate AfcR. The plan demonstrates the program’s commitment to 
evaluating applications for additional AfcR candidates, and implementing these candidates where 
safety, productivity, and life-cycle cost benefits can clearly be provided. 

A high-leverage prototyping program as described in section A would be of particular value to the 
S.S. Freedom Program. During the fall of 1988, the S.S. Freedom Program at Level II initiated a high- 
leverage prototyping program via an informal request to the contractor work package centers for high- 
leverage prototyping concepts. In response to the Level II request, a variety of ideas and concept 
studies were received through three Level II working groups: the Advanced Automation Working 
Group (AAWG), the Robotics Working Group (RWG), and the Artificial Intelligence and Expert 
Systems Technology Working Group (AIESTWG). These ideas and concepts constitute the set of 
high-leverage prototype A&R candidates to be evaluated. Thus, a need was established for a process 
and method to evaluate the alternative high-leverage prototype A&R candidates. The process, the 
methodology, and results of the evaluation are the subject of this report. 

C. SUMMARY 

The study conclusions and recommendations are summarized below: 

1. The Process 

(1) The first implication is that high-leverage prototyping is beneficial to the Space Station 
Freedom Program as a means for transferring technology from the advanced development 
program to the baseline program. Such a mechanism serves as a formal bridge between 
technology development programs and end-users. The bridge, however, needs to be 
specified with greater detail. 

(2) The purpose of high-leverage prototype A&R candidates is to provide near-term, low-risk 
AfcR applications for inclusion in the baseline S.S. Freedom . The interest in low-risk 
development for this application is to make the application part of the near-term baseline. 
High risk developments could also be considered if evolutionary applications are included. 
Any high-leverage prototyping RfcD program should determine whether to subsidize high- 
value, low-risk developments or high-value, high-risk developments. The needs in both 
cases are different. 

(3) The application of multiattribute decision analysis methods was useful for meeting the 
numerous requirements and constraints specific to the S.S. Freedom Program. In 
particular, the ability of the methodology to address multiple attributes with uncertainty, 
multiple interested parties to the decision, and monitoring of consensus was seen to be 
effective and practical. 
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(4) The process would be facilitated by computer-based interactive programs to enter and edit 
the evaluation problem to the extent that major steps could be automated. Such a 
Decision Support System (DSS) for use in a distributed fashion by individuals or as a 
central display for decision conferencing by a group would allow greater flexibility for 
examining alternative assumptions. Similar software could also be developed for 
conducting the argumentation analysis of the technical assessment. 

2. The Application 

(1) Promising technology areas of potential relevance to the S.S. Freedom are: 


( 2 ) 


( 3 ) 


( 4 ) 


DIA* 

STD 

KBS* 

EXT 

AUT* 

FAU* 

MNT* 

ROB* 

PRO* 


Diagnostic Expert System for electric power and data management 
Standards and Tools for Expert Systems 

Knowledge- based system for fault detection in data management and 

operations management systems 

Testbed for extended fault-tolerant testing 

Study of robotics for assembly and fasteners with testbed 

Knowledge- based system for electric power subsystem fault prediction 

Study of knowledge-based system development and maintenance tools 

Software upgrades to demonstration of robotic control of assembly 

Prototype crew scheduler using advanced search methods 


The results of this study are supported by the separate working group evaluations although 
a number of serious differences were observed. These differences may be attributable to 
the use of primarily technical criteria as opposed to the Level II S.S. Freedom Program 
criteria outlined in this study. 

The majority of recommended high-leverage prototyping A&R candidates involve the 
study or development of software either for specific applications to systems or to support 
the software development process in some manner. The high values for software-oriented 
applications may be a reflection of two world views: (i) high-leverage prototyping should 
encourage high-risk investments and software developments are perceived as risky; or (ii) 
high-leverage prototyping should encourage low-risk investments to increase the potential 
for inclusion in the Baseline S.S. Freedom and software developments are perceived as low- 
risk. It is not clear whether or how these world views might have affected the types of 
high-leverage prototype A&R candidates at the point of submittal. 

The applications of the recommended high-leverage prototyping A&R candidates focus 
primarily on fault detection, fault tolerance, and fault prediction software using Expert 
Systems and Knowledge-Based Systems. The target subsystems for these applications are 
the electric power subsystem, the data management subsystem, and the operations 
management subsystem. 


^Possible overlap with existing work packages and/or with other high-leverage prototyping A&R 
candidates. 
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3 . 

( 1 ) 

( 2 ) 

(3) 


(4) 


( 1 ) 

( 2 ) 

(3) 

(4) 


The Model Results 

The high-leverage prototype AkR candidates examined exhibit high degrees of uncertainty 
based in part, on schedules and deliverables. Lack of detailed proposal data is likely to 
have had some, albeit non-quantifiable, effect. 

The use of probability distributions is helpful for aggregating and quantifying the 
magnitude of uncertainties. 

The use of argument analysis, although time-consuming, proved useful for expressing the 
underlying patterns of reasoning for the numerical estimates of the attribute states. The 
numerical estimates provided a measure of the uncertainty whereas the argument analysis 
provided a window for the credibility of the estimates. The process used would be aided 
considerably by a computer-based interactive system. 

There is a high degree of concordance (agreement) in the rankings at both the individual 
and group levels. There is similar robustness under different decision model assumptions. 
This is due to the comparable weightings assigned to the evaluation attributes by the 
interested parties interviewed. 

Recommendations 

A high-leverage prototyping program should require the establishment of common goals 
and perspectives prior to the call for concepts. An element of setting such goals would be 
the development of program criteria to facilitate the collection of more complete 
information using a common format. 

If a high-leverage prototyping program is initiated, a formal call for proposals should be 
accompanied by a prescribed set of programmatic AfcR evaluation attributes and a 
common response format, to the extent feasible. 

Establish an on-going AfcR function at the systems engineering and integration level to 
support programmatic evaluations and systems engineering trade studies for the 
S.S. Freedom Program. Such a function has never been formally established within the 
S.S. Freedom Program. 

Develop and implement a concrete plan for AfcR/new technology implementation and 
incorporation within the S.S. Freedom Program and constitute the plan into an operational 
process. 


D. REPORT ORGANIZATION 

This report consists of seven sections. Section I introduces the purpose, background, and 
summary of the study. Section II presents a description of the AfeR implementation process. Section 
III describes the methodology for evaluating AfcR candidates in both detail and summary form. 
Section IV illustrates the methodology with an application to a set of twenty-one high-leverage 
prototype AkR candidates. Section V presents the results of the application followed by a discussion 
and the conclusions in Section VI. References are listed in Section VII. 
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SECTION II 

AUTOMATION AND ROBOTICS IMPLEMENTATION 


A. AfcR IMPLEMENTATION PROCESS 

The benefits of automation and robotics to the S.S. Freedom Program will come from 
applications responding to mission and operational needs. AfcR technologies based on such needs hold 
promise for enhancing safety, productivity, and mission success, and reducing life-cycle costs and risks. 

An idealized AfcR implementation process is shown in Figure 2-1. Requirements are derived from 
the program objectives, i.e., the mission. Functions to be performed during the design, development, 
testing, evaluation, assembly, operations, and evolution phases are then derived from the requirements. 
A first-level screening of the functions is then performed to obtain a list of candidate functions that can 
be accomplished using AfcR technologies. These functions can be described in terms of performance 
requirements and physical constraints. 

The technology base is the source of AfcR design candidates. The technology base includes 
candidates from NASA RfcD programs, industry, Department of Defense, and- academia. The design 
candidates can be described in terms of their performance capabilities and physical characteristics. A 
technical evaluation process matches the appropriate task with the appropriate design candidate or 
results in task redefinition or refinements in design. This process ultimately results in a set of AfcR 
candidate technologies that are capable of performing particular tasks. 

A program-level evaluation then takes place in which the candidates are evaluated according to a 
set of program-approved criteria. These criteria should include, at a minimum, safety, productivity, 
life-cycle cost, and technical risk (i.e., maturity), but can include as many others as are warranted (see 
Section IV). A decision package on the most promising candidates is then prepared for S.S. Freedom 
Program management, which has several options for disposition of the candidates. If the candidate is 
technologically mature or available off-the-shelf, it can go directly into baseline implementation if 
funding is allocated. If a candidate has a higher degree of technical risk or technological immaturity, 
but may have high pay-offs, it is an ideal candidate for a prototyping program. Other candidates may 
be more suitable for a future growth phase of the S.S. Freedom Program, or should be deferred until 
funding is available. 

Some “candidates” are already funded in the baseline (i.e., the FTS). In a case such as this, the 
evaluation process can be used to identify tasks for the FTS. 

B. AfcR CANDIDATE DOMAINS 

Areas of the program to which AfcR technologies can be applied include the broad domains of (I) 
accommodations and standards for A&R, including design guidelines for the baseline Space Station and 
hooks and scars for evolution; (2) applications, which includes baseline content, targeted additions to 
the baseline, high-leverage prototyping of promising candidates, and evolution candidates; and (3) the 
development environment, which includes testbeds and software development support. 

The applications domain refers to the use of A&R technologies for the performance of Space 
Station tasks in the assembly and operations phases, for both ground and on-orbit systems. Some 
applications are already funded in the baseline program; other applications may be added as a result of 
the evaluation process described above. _ 
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Figure 2-L A&R Implementation Process 











Automation and Robotics Implementation 


The integration of AfcR candidates into the Space Station design must consider (1) design 
standards for implementation during the design phase to ensure commonality and conformance of 
design philosophy, and (2) the inclusion of hooks and scars for evolution to enable the Station to 
accommodate Advanced A&R during the operational phase. The inclusion of hooks and scars for 
evolution is an activity focused on the period of mature Station operations. The development and 
monitoring of design standards is an activity focused on the present. The standards must be in place as 
early in the design process as feasible. Because accommodations and standards span a long project 
lifetime, the complexity of the systems engineering effort to install such considerations is significant. 

The development environment is also an important area for A&R. Development environment 
technologies are tools that can improve the development, performance, manufacture, and checkout of 
hardware and software by improving the development environment. These design tools can provide 
additional safety and level of confidence in station technologies prior to transferring the technologies to 
the actual ground or space application. Examples include engineering and manufacturing processes, 
various modeling applications for the design phase, automated testbeds for demonstrating and testing 
systems and components, design workstations, and software tools for verification and validation of 
software. 

C. AfcR EVALUATION CRITERIA 

The PRD requires A&R candidates to be evaluated on the basis of safety, productivity, and life- 
cycle cost benefits. Other benefits to be provided such as growth potential and spinoff to U.S. industry, 
have been identified by previous studies for the S.S. Freedom Program during Phase B (Reference 3). 
These evaluation criteria formed the basis for development of a hierarchy of objectives and attributes 
described in Section IV. 
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SECTION III 

METHODOLOGY FOR AkR CANDIDATE EVALUATION 


This section describes and illustrates the general methodology used to evaluate and compare 
alternative A k R candidates. The methodology consists of a number of steps that characterize the 
alternative AfcR candidates, assign utility values to the alternatives, and rank the alternatives based 
on these utilities. 

The general evaluation methodology may be summarized as follows. The process begins with the 
selection of a set of descriptive but quantifiable attributes designed to characterize each A&R 
candidate. Values for this set of attributes are then generated for each alternate AfcR candidate that 
specify its response (e.g., cost, performance, safety) under a specific set of assumptions regarding design 
options and operating environments. (The attributes are discussed in Section IV.) A decision tree can 
be constructed to relate economic, technological, and environmental uncertainties (i.e., the operating 
environment) to the cost and performance outcomes (i.e., attribute values) of the alternative A&R 
candidates. Multiattribute utility functions that reflect the preferences and perceptions of 
knowledgeable individuals are generated based on interviews with selected personnel. The functions are 
then employed to generate a multiattribute utility value for each AfcR candidate, based on its 
characteristics under the scenarios reflected within the decision tree. The decision tree is used to 
compute an expected multiattribute utility value for each alternative AfeR candidate, the expected 
value being taken over the attribute probability distributions. Alternative A&R candidates are ranked 
according to this expected multiattribute utility value and analyzed for parameter, model, and group 
sensitivities, risk, and concordance (agreement among rankings). 

A. MULTIATTRIBUTE DECISION ANALYSIS 

Multiattribute decision analysis is a methodology for providing information to decision makers for 
comparing and selecting from among complex alternative systems in the presence of uncertainty. The 
methodology of multiattribute decision analysis is derived from the techniques of operations research, 
statistics, economics, mathematics, and psychology. Researchers from a wide range of disciplines have 
participated in the development of multiattribute decision analysis. The first books and papers on the 
subject appeared in the late 1960s (References 4, 5, 6, 7). The most practical, extensive, and complete 
presentation of an approach to multiattribute decision analysis is given in the 1976 work of Keeney and 
Raiffa (Reference 8). Although several approaches to multiattribute decision analysis have been 
developed (References 9, 10, 11, 12, 13), the method used in this study corresponds to an abbreviated 
form of that of Keeney and Raiffa. The assumptions needed for the abbreviated form are discussed 
herein. 

Decision problems involving the preference ranking of alternatives, whatever the specific 
methodology, require two kinds of models (Figure 3-1). One is a “system model and represents 
the alternative systems (including any uncertainties) under consideration. The other is a 
“value model* and represents the preference structure of the decision makers whose preferences are 
being assessed. The system model describes the alternative systems available to the decision makers in 
terms of the risk and possible outcomes that could result from each. Risk arises from the uncertainty 
associated with each alternative and from the uncertain environment in which the alternative 
exists. The outcomes describe the possible consequences of the alternatives. 
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Because of the element of risk, the selection of a specific alternative does not in general 
guarantee a specific outcome, but rather results in a probabilistic situation in which only one of 
several outcomes may occur. These outcomes, with their measurable attributes, then form the 
input to the value model. 

The value model prioritizes the outcomes in terms of the preferences of the decision makers for 
the various outcomes. The measurable attributes of the outcomes are aggregated in a formula (called a 
multiattribute utility function) whose functional form and parameters are determined by the preference 
structure of the decision makers. The output of the value model is a multiattribute utility 
function value for each outcome (outcome utility). These outcome utilities are entered back into 
the system model where the utility of an alternative can be calculated by taking the expected 
utility value of the outcomes with each alternative system. These expected utilities for each 
alternative define a preference ranking over the alternatives, with greater expected utilities being more 
preferred. 

This section describes a procedure for computing the transformation of uncertain alternatives and 
their attribute states into a probabilistic measure of preference using multiplicative and additive multi- 
attribute utility functions for the transformation of random variables. Given a model of preference for 
prioritizing the attribute states, the transformation of random attribute state variables is used to 
obtain a probabilistic ranking of the alternatives. Thus, not only is the overall expected utility 
obtained, but also its distribution, mean and variance. A discussion of the multiattribute decision 
analysis methodology is presented in deterministic form and then extended to the probabilistic form. 

1. Decision Trees 

Decision trees are used to represent the system model and the inputs to the system model 
at the gross level required for the decision analysis. Decision trees are graphically depicted by decision 
nodes (represented by squares), with alternative paths emanating from them and by chance nodes 
(represented by circles), with probabilistic paths emanating from them. All paths either terminate at 
another node or terminate at an outcome, which is a description of the consequence of traversing a 
specific set of paths and nodes through the decison tree from beginning to end. There can be only one 
originating node (either a decision node or a chance node). There can be many outcomes terminating 
the decision tree, depending on the complexity of the decision tree. 

Figure 3-2 shows a typical decision tree, terminating in 10 outcomes. The symbols “Dj” stand for 
the ith decision node (“D” for decision). The symbols “Pj” stand for the jth chance node (“P” for 
probabilistic). The symbols “C^” stand for the kth outcome (“C” for consequence). Every path 
emanating from a decision node corresponds to an alternative that the decision makers can select, 
where “A-£” stands for the I th alternative selected at the ith decision node. The decision makers can 
select one and only one path at each decision node. Every path P« emanating from a chance node 
corresponds to one of the uncertain and uncontrollable chance states that can occur at that node, where 
p* m is the probability that the mth chance state will be realized at the jth chance node. The Pj m s 
must obey the laws of probability theory. Thus, one and only one chance path can be realized from a 
chance node and the Pj m s must sum to 1.0. 

The chance nodes and their associated chance paths and probabilities are called “gambles’ 1 or 
“lotteries’ 1 in the literature. This report shall refer to them as gambles. An example of a gamble 
would be a flip of a coin, which could be expected to come up heads 50% of the time and tails 50% of 
the time. 
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Graphically, such a gamble would be displayed as: 

HEADS 

0.50 / 

°( 

0.50 \ 

TAILS 

Figure 3-2 has an example of every kind of node-path-outcome relationship. There are examples 
of decision-node to decision-node paths, decision-node to chance-node paths, decision-node to outcome 
paths, chance-node to decision-node paths, chance node to chance-node paths, and chance-node to 
outcome paths. 

As an example of how the decision tree might be traversed, imagine that the decision maker 
selects Alternative Path A^ at Decision Node D, , where he must start. This leads to Chance Node P ^ 
where Chance Path P^ is realized, leading to Chance Node P^, where Chance Path Pg 2 1S realized, 
and terminates with Outcome C^q. 

2. Determination of Probabilities 

The decision trees have probabilities associated with all of the chance paths. these 
probabilities need to be assessed as perceived by the decision makers or as perceived by experts whose 
judgment the decision makers would be willing to accept. 

Two conditions must be satisfied by the probabilities associated with the chance paths emanating 
from 'the single-chance node. The probabilities must be “coherent” and “veridical” (Reference 14). To 
be “coherent” means that the probabilities obey the laws of probability theory. This requires that the 
chance paths emanating from a single-chance node correspond to probability events that are mutually 
exclusive and collectively exhaustive (one and only one of the chance paths must occur), and that the 
probabilities assigned to all the chance paths emanating from a single-chance node must be non- 
negative and sum to 1.0. To be “veridical” means that the probabilities must bear some correspondence 
to reality. For example, if the probability 1/n were assigned to the “n” chance paths emanating from a 
chance node, coherence would be satisfied if the chance paths corresponded to mutually exclusive and 
collectively exhaustive events because these probabilities sum to 1.0. However, veridicality would be 
violated if one of the chance paths was perceived as being very improbable, because the assignment of a 
probability of 1/n to that chance path would be inappropriate. 

An excellent review and an extensive bibliography on the assessment of probabilities is given in 
Hogarth (Reference 15). The philosophy and practice used in probability assessment by the Decision 
Analysis Group at SRI, International is given in Spetzler and Stael Von Holstein (Reference 16). 
Elementary discussions of probability assessment are given in Brown, Kahr, and Peterson; Keeney and 
Raiffa; and Schlaifer (see References 14, 8, and 7, respectively) and Winkler (Reference 17). The 
probability assessment technique presented in this report attempts to satisfy the requirements of 
coherence and veridicality with a minimum of effort on the part of the assessor and the decision makers 
or experts whose subjective probabilities are being assessed. 

The probability assessment technique first involves the construction of chance paths satisfying the 
mutually exclusive and collectively exhaustive condition. In Figure 3-2, three chance paths (Pj^, ^12’ 
and P n ) emanate from Chance Node P^. These three chance paths might, for example, correspond to 
the events: (1) For Chance Path P n , Alternative AkK candidate A 12 costs $25,000 or less, with a 
most probable cost of $15,000 and performs as specified; (2) for Chance Path Pj 2 , Alternative A&R 
candidate A^ 2 costs more than $12,000 with a most probable cost of $18,000 and performs as specified; 
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and (3) for Chance Path P ^ Alternative AfcR candidate A^ has a most probable cost of $10,000 but 
does not perform well enough to be used. Rigorously, according to decision analysis theory, “certainty 
equivalent” costs (see Reference 8) should be used rather than “most probable” costs as in the 
preceding statements, but for this discussion “most probable” will suffice. 

The next step is to assess probabilities to be assigned to each of the chance paths. This is done 
by interviewing either the decision makers or their designated experts according to the following 
format: 

(1) Ask the interviewee to rank the chance paths emanating from a chance node in order of 
decreasing perceived probability of occurrence. 

(2) For the chance node, ask the interviewee, “How much more probable is the most-probable 
chance path than the least- probable chance path? A little? Ten times? A hundred 
times?” 

(3) If the reply is a number, such as “six times more probable,” then consider the next least- 
probable chance path. 

(4) If the reply is “a little,” then ask, “Is the most-probable chance path 10%, 25%, or 50% 
more probable?” The interviewee should respond with whatever percentage he feels is 
appropriate. Then consider the next least-probable chance path. 

(5) Repeat (2) to (4) for all of the chance paths of the chance node. 

(6) Repeat (1) to (5) for all of the chance nodes relevant to the interview. 

This is all the information that is required from the interview for assessing probabilities for the 
chance paths. The probabilities for the chance paths can be calculated from the interview responses by 
solving a set of simultaneous equations of the form: 


p. * = x , o p , 
J rn jrr/ jit/ 
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The solution is: 


p n = 0.526; p 12 = 0.421; p 13 = 0.053 


3. Objectives Hierarchy 

The outcomes that terminate the decision tree are to be described in terms of an objectives 
hierarchy that (1) expresses the preference structure of the decision makers, and (2) is constructed in a 
manner compatible with the quantification and mathematical conditions required by a multiattribute 
utility function of the value model. The objectives hierarchy expresses the preference structure of the 
decision makers in ever increasing detail as one proceeds down through the hierarchy from overall 
objective to a lower-level hierarchy of sub- objectives. Below the subobjectives are “criteria.” The 
criteria must permit the quantification of performance of the alternatives with respect to the sub- 
objectives. Associated with each criterion is an “attribute,” a quantity that can be measured and for 
which the decision makers can express preferences for its various states. Figure 3-3 shows an objectives 
hierarchy with the associated attributes. 

The set of attributes must satisfy the following requirements for the value model to be a valid 
representative of the preference structure of the decision makers: 

(1) Completeness: The set of attributes should characterize all of the factors to be considered 
in the decision-making process. 

(2) Comprehensiveness: Each attribute should adequately characterize its associated criterion. 

(3) Importance: Each attribute should represent a significant criterion in the decision-making 
process, at least in the sense that the attribute has the potential for affecting the preference 
ordering of the alternatives under consideration. 

(4) Measurability: Each attribute should be capable of being objectively or subjectively 
quantified; technically, this requires that it be possible to establish an attribute utility 
function for each attribute. 

(5) Familiarity: Each attribute should be understandable to the decision makers in the sense 
that they should be able to identify preferences for different states of the attribute for 
gambles over the states of the attribute. 

(6) Nonredundancy: Two attributes should not measure the same criterion, thus resulting in 
double counting. 

(7) Independence: The value model should be so structured that changes within certain limits 
in the state of one attribute should not affect the preference ordering for states of another 
attribute or the preference ordering for gambles over the states of another attribute. 
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Figure 3-3. Hierarchy of Objectives, Criteria, and Attributes 
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4. Attribute Utility Functions and the Multiattribute Utility Function 

The set of attributes associated with the objectives of a decision analysis must satisfy 
certain measurability and mathematical requirements. If these requirements are satisfied, then it is 
possible to formulate a mathematical function, called a multiattribute utility function, that will assign 
numbers, called outcome utilities, to the set of attribute states characterizing an outcome. The 
multiattribute utility function used is that of Keeney and Raiffa (see Reference 8). The outcome 
utilities generated by the Keeney and Raiffa multiattribute utility function have the properties of Von 
Neumann and Morgenstern utilities (Reference 18), that is: 

(1) Greater outcome utility values correspond to more preferred outcomes. 

(2) The utility value to be assigned to a gamble is the expected value of the outcome 
utilities of the gamble. 

The mathematical axioms that must be valid for these two properties to hold were first 
derived by Von Neumann and Morgenstern (Reference 18) and expositions of these axioms are 
given in a variety of sources (References 19, 20, 21, 22). 

To every outcome an N-dimensional vector of attributes x = <Xp..., x^> will be associated, 
the set of which satisfy the attribute requirements cited above. Most of the attribute requirements are 
self-evident. One requirement, that of attribute independence, is a condition that makes it possible to 
consider preferences between states of a specific attribute, without consideration of the states of the 
other N-l attributes. It is thus possible to construct an attribute utility function that is independent 
of the other attribute states, and which, like the outcome utility function, satisfies the Von Neumann 
and Morgenstern properties for utility functions. This condition of independence, or some equivalent 
mathematical condition (see Reference 8 for alternative formulations), is necessary for the Keeney and 
Raiffa methodology. It is necessary to verify that this condition is valid in practice, or more correctly, 
to test and identify the bounds of its validity. 

To continue the discussion from this point, it is necessary to introduce some mathematical 
notation: 

Xn — The state of the nth attribute. 

Xn = The least-preferred state to be considered for the nth attribute. 

x* “ The most-preferred state to be considered for the nth attribute. 

x = Vector of attribute states characterizing a specific outcome: x = <x^ , X2 , . - x^> 

x° = An outcome constructed from the least preferred states of all the attributes: 


y° — /v° v° y° ^ 

X — <.X| , X2 J • • M 


= An outcome constructed from the most preferred states of all attributes: 


(x n , *£ 


u n ( x n ) 
U(x) 

K 


X* = <x* x£, . . xt> 


= An outcome in which all attributes except the nth attribute are at their least- 
preferred state. 

= The attribute utility of the nth attribute. 

= The outcome utility of the outcome x (vector of attribute states). 

= The attribute scaling constant for the nth attribute: k n = u ( xj , Xn J 
= The master scaling constant for the multiattribute utility equation. It is 
algebraic function of the k n values. 
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The mathematics permit the arbitrary assignments: 

u n( x n ) = 0*0 (least preferred) and u n (xn ) = 1.0 (most preferred) 

Thus, the attribute utility function values will range from 0.0 to 1.0. Attribute utility function 
values for attribute states intermediate between the worst and best values are assessed by determining 
a value of p n such that the decision makers or their designated experts are indifferent between receiving 
x n for sure, or, a gamble that yields x£ with probability p n or x£ with probability l-p n . Graphically, 
assess p n , so that: 


Pn j x n 

x n ~ 

1 - p n \ Xq 

where means indifference. It follows from the mathematics that: u n (x n ) = p n . This indifference 
relation is repeated for various attribute states until either a continuous utility function can be 
approximated or enough discrete points have been assessed for the attribute states under consideration 
in the analysis. 

A similar approach is used to assess the scaling constants (weights). A value for k n is assessed 
such that the following indifference holds: 



The Deterministic Case 

The deterministic case refers to the calculation of a single numerical measure for outcome utility 
that assumes the attribute state values do not represent uncertainty in any way (with a probability 
distribution). With the assessment of the single-attribute utility functions and attribute scaling 
constants, the multiattribute utility equation can be solved to yield a deterministic outcome utility 
value for any outcome under consideration. The multiattribute utility function can be stated in one 
of two forms. The first form is the multiplicative multiattribute utility function: 


N 

f N 

] 

If, J2 k n * L0 

, then: => U(x) = 17 \ TI l 1 + K ' k n * u n( x n) 1 ~ 1 

\ Eq 1 

n=l 

U=1 

J 

where the master 

scaling constant, K, is solved from the equation: 
1 + K = ft ( 1 + K* k n 1 

n=l 



The second form is the additive multiattribute utility function: 


If, 


N 


X! kn — 

n = 1 


1.0 


N 

then: => U(x) = kn ' 

n=l 


Eq 2 


u n( x n) 
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The outcome utility function values, like the attribute utility function values, will all range from 
0.0 to 1.0 with: 


U n (x° ) = 0.0 (most preferred) 

U n (x*) = 1.0 (least preferred) 

Although the mathematical equations appear complex, they can be easily solved, and the 
information required in the interviews with the decision makers can be minimized. An extended 
discussion of these equations, their solution, and the assessment of the required data, together with 
examples taken from actual applications, is given in Keeney and Raiffa (see Reference 8). 

The discussion in this section uses an abbreviated form of Keeney and Raiffa’s methodology to 
reduce the interview (questionnaire) time for the interviewee. An assumption is made that 
utility independence of each attribute implies pair-wise utility independence (i.e., the attributes 
exhibit utility independence when taken two at a time). This assumption allows the use of 
Formulation (4) of Theorem 6.2 of Keeney and Raiffa (see Reference 8). Given single-attribute 
utility independence, it is difficult to construct a realistic example where pair-wise utility 
independence would be violated. 

The Probabilistic Case 

The probabilistic case as defined here involves a transformation of random variables where 
the random variables are the attribute states ( x = <x^, —i x ]^^)i transformation can either be 
the additive or multiplicative model, and the result is a cumulative distribution function of the 
outcome expected utility. Thus a decision maker can view how various levels of uncertainty in the 
attribute states contribute to the overall uncertainty of each alternative. In fact, depending on the 
situation, a decision maker may choose an alternative with a lower expected utility— and lower 
uncertainty over an alternative with a higher expected utility, but with greater uncertainty. This 
approach enables such a choice. 

Because of the non-linear form of the multiplicative model, the analytical transformation of an 
arbitrary number of attributes is intractable. However, Monte Carlo simulation can be used to 
effectively compute the transformation. At each trial, a random sample is drawn from the attribute 
state CDFs (the x: n ’s). The sequence of these random values, x a (<* = 1, 2, ..., M = number of Monte 
Carlo trials), is then substituted into the individual attribute utility functions to compute a 
probabilistic attribute utility value. The u n ( x n )’s) are then used to compute an overall probabilistic 
expected utility of the alternative, U(x) (Figure 3-4). 
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Figure 3-4. Probabilistic Simulation for Attribute, n, and Monte Carlo Trial, a 


Formally, the CDF’s for the attribute states, x n , are denoted F n (x n ). Introducing the new notation, 
the probabilistic mulitiplicative case becomes: 



and the probabilistic additive multiattribute utility function: 


N 

N 


If k n = 1.0 , then: => 

0(x) = k n • u n (x n ) 

Eq 4 

n=l 

n=l 
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The outcome utility function values, like the attribute utility function values, will range from 
0.0 to 1.0 with: 

U(x°) = 0.0 (least preferred) 

U(x*) = 1.0 (most preferred) 

A major assumption required to use this technique is that of probabilistic independence between 
the attribute CDF’s. Whether or not this condition is met depends on a number of factors. For 
example, in general the ranges of the attributes are narrower than 0 -► oo which tends to ease both the 
probabilistic and preference/utility independence requirements. In any case, this assumption ought to 
be examined carefully. Note also that the attribute scaling constants, and utility functions are fixed— 
that is, the preference model is deterministic — the uncertainty in the attribute states (the alternatives) 
is the only probabilistic quantity. Further work is needed to understand the uncertainties in preference 
(perceptions of the attribute ranges). 

The steps needed prior to ranking the alternatives are: identification of alternatives, definition 

of the decision via an objectives hierarchy, the quantification of the criteria in terms of measurable 
attributes with either point estimates or cumulative distribution functions (CDFs), and determination 
of a multiattribute utility function with attribute utility functions and attribute scaling constants 
corresponding to the preference structure of the decision makers. The ranking of the alternatives 
proceeds in one of two ways. 

The deterministic case uses the attribute states (if a CDF is present, the attribute state means 
are used) to compute the multiattribute utility function to calculate outcome utilities for all of the 
outcomes. Because point estimates are used, this is a one-time calculation for each alternative. The 
resulting rankings do not reflect consideration of uncertainty, they are deterministic-determined by 
the single valued attribute states. 

The probabilistic case simulates the uncertainty in attribute states by performing a 
transformation of random variables using the additive or multiplicative decision model (the 
overall expected utility is the dependent variable and the attributes are the independent random 
variables). Monte Carlo simulation is used to repeatedly sample from each attribute state CDF and 
compute a running mean and standard deviation for the distribution of expected utility. The 
means and standard deviations provide information on how uncertainties in the attribute states 
are passed through to the rankings. 

5. Ranking the Alternative A&R Candidates 

The steps needed prior to ranking the alternatives are: the development of a decision tree, 
the determination of probabilities for the decision of an objectives hierarchy, the quantification of the 
criteria in terms of measurable attributes, and the determination of a multiattribute utility function 
with attribute utility functions and attribute scaling constants corresponding to the preference structure 
of the decision makers. The ranking of the alternative A&R candidates proceeds as follows (Figure 3- 
5): 

(1) Use the multiattribute utility function to calculate outcome utilities for all of the 
outcomes of the decision tree. 

(2) Calculate a utility value to be assigned to all chance nodes by taking the expected utility 
value of the utilities assigned to the termination of the chance paths of the chance nodes. 
The chance paths may terminate at outcomes, other chance nodes, decision nodes, or a 
combination of these. 
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(3) Calculate a utility value for all decision nodes by selecting the decision path that 

terminates in an outcome, chance node, or decision node with the highest utility value. 

The utility value of that path shall be the utility value assigned to the decision node. 

The decision tree for this study has an originating decision node whose decision paths correspond 
to the alternative A&R candidates under consideration. Steps (1) through (3) are performed by 
starting with the outcomes shown in Figure 3-2 and assigning utility values to these outcomes. Then 
Steps (2) and (3) are performed by a “folding back” process, proceeding from right to left, and 
assigning utility values to the chance nodes and the decision nodes. Finally, utility values are assigned 
to the decision paths emanating from the originating decision node on the left. These utility values are 
the ones assigned to the alternative AkR candidates. Because greater utility values correspond to more 
preferred A&R candidates, a rank order in preference for the alternative A&R candidate can be 
assigned in correspondence with the utility values. A quantifiable and tangible measure of the strength 
of preference between the alternative AkR candidates can be obtained by referencing each alternative 
AkR candidate to a set of AkR candidates where only one attribute, such as initial cost, is varied 
(References 25 and 26). The differences in the attribute states of this one attribute varied in order to 
obtain indifference to each of the alterative A&R candidates will provide a tangible measure of the 
strength of preference between the alternative A&R candidates. 

6. Group Decision Models 

This section reports on a technique that has proven useful for a number of applications of 
multiattribute decision analysis involving multiple and disparate groups as stakeholders. The decisions 
involved similar ranking of technologies for the purposes of further research and development. 

This section does not report on techniques involving multiple computer terminals, decision rooms, 
interactive graphics, and other similar hardware for facilitating the process . Rather, this paper 

describes workable techniques for examining group agreement incorporated into the decision process. 

The group decision making approach used here is not new— comparing the judgments of 
individuals and the resulting rankings has been an area of ongoing study for some time. However, the 
manner and extent to which these techniques have been used has proven useful in this study. 

The approach was developed to examine group preferences in a multiattribute decision making 
context. When performing a simple (i.e. one group) multiattribute decision analysis, there is the 
problem of determining whether multiple rankings are in general agreement for the group as a whole. 
Two general approaches for evaluating group agreement are described: (1) examine the individual 

rankings directly to deduce agreement; or (2) use the individual rankings to compute a group ranking. 
An approach to both methods is described here. 

In many situations there is no single, isolated, decision maker. When one individual holds the 
ultimate responsibility for a decision, this person may elect to delegate the decision-making 
responsibility to a group, or at least consider the preferences of several others prior to making the 
decision. 

Unfortunately, there presently exist no analytical models for group decision making that do not 
violate some intuitively desirable conditions. Arrow (Reference 23) has demonstrated this fact. 
Extensive discussions of group decision making can be found in Fishburn (Reference 24), Luce and 
Raiffa (see Reference 15), and Sen (Reference 25). What can be done is to look at a range of group 
decision models, and where consensus of the models is found, define that as the consensus of the group 
(Reference 26). 






Three candidate rules considered here are the Rank Sum Rule, the Nash Bargaining Rule, and the 
Additive Utility Rule. 

The Rank Sum Rule requires the calculation of the sum of the ordinal ranks for each alternative, 
with the alternative receiving the lowest rank sum being most preferred (References 23 and 27). 

The Nash Bargaining Rule calculates the product of the utilities assigned by all the individuals to 
an alternative (Reference 28). The alternatives with the greater utility product are more preferred, and 
from this a group preference order can be established. The Nash Bargaining Rule satisfies Nash’s four 
axioms of “fairness.” As the number of judges increases, the Nash utilities decrease because the 
individual utilities are < 1. Hence, for even ten decision makers, the Nash utilities are small. Without 
loss of generality, the Nash utilities can be rescaled by taking the nth root of the product of the 
individual utilities, where n is the number of decision makers in the group. 

The modern formulation of the Additive Utility Rule is that of Harsanyi (Reference 29). The 
Additive Utility Rule averages the utility values assigned by the individuals to each alternative, with 
higher average utility values being more preferred. 

It should be reemphasized that there is no theoretically compelling reason to use the results of any 
of these group decision rules, but they do provide information concerning the collective preferences of 
the decision makers. What is valuable is that techniques are available to test the agreement between 
the decision rules. That is, while no singular method may suffice for the aggregation of individual 
preferences, by examining a variety of rules under different assumptions, insights regarding the decision 
can be viewed in a controlled manner. Smith and Feinberg (Reference 30) have developed a number of 
guidelines for analysis of group judgments that are used in the present study: 

Guideline 1: Combining group judgments and preferences can provide “indicators” of general 

agreement and whether the group is moving toward or away from general 
agreement; however, it is not decisive. If two integer ranks are close (indicating 
possible indifference) but the rankings are reversed, there will be a slight penalty in 
concordance. 

Guideline 2: Studies with identifiable, conflicting groups provide a way to understand the 

differing world views of the participants’. This is one of the valuable contributions 
of examining group agreement— discordance initiates a search for these assumptions. 

Guideline 3: Always maintain the individual rankings within the analysis; once the results are 

computed at the group level alone, a degree of freedom is lost and the credibility of 
the analysis suffers accordingly. 

Guideline 4: Groups are often (but not always), defined by institutional or organizational 

boundaries. 

Guideline 5: Always be prepared to explain why the rankings of two groups agree- especially if 

there is a perceived conflict (corollary: it is generally easier to explain why they 
disagree). Agreement between dissenting groups can sometimes be attributed to 
normative versus descriptive behavior (e.g., “safety” is an important factor to 
everyone until it is discovered that it leads to the same technical conclusion as a 
competing organization). 
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Guideline 6: Combining and comparing group judgments is a useful way to search for patterns of 

agreement in situations with numerous and diverse groups and although it may or 
may not be used to affect agreement between groups, it provides a tangible measure 
of progress during the evolution of the project or negotiations. 

Again, it should be reemphasized that there is no theoretically compelling reason to use the 
results of any of these group decision rules, but they do provide information concerning the collective 
preferences of the decision makers. 

B. RISK ANALYSIS 

Another element of the sensitivity analysis effort is that of risk analysis. Because of the close 
connections between the notions of risk and uncertainty, this subsection explains and illustrates the 
elements of both risk analysis and uncertainty and describes how these concepts are incorporated into 
the multiattribute decision model and into the sensitivity analysis. 

1. Uncertainty and Risk Defined 

Definitions of uncertainty and risk typically involve the concept of probability and this 
commonality often leads to confusion about whether there is any difference. The riskiness of a 
technology development usually refers the the probability that a technology can actually be produced 
and the probability of success is an uncertainty. The distinction between uncertainty and risk as used 
in this study is given below: 

Uncertainty is defined as a numerical estimate of probability associated with a random variable, 
x— in this study, the random variable x is an attribute state variable and the probability is the chance 
(discrete case) or probability distribution of x occurring. For example, the uncertainty in operations 
cost could be represented with the statement, “The probability that initial cost for A&R candidate 
number 3 is greater than $250,000 is 0.90.” 

Risk is defined as a combination of a probability and an associated event consequence. In many 
ways, statements about uncertainty are also statements about risk, however, the notion of risk carries a 
connotation of “something to be avoided” with it. The “something to be avoided” is the consequence 
of the event occurring. For example, the risk of an A&R technology could be represented by the 
statement, “The probability that initial cost for AkR candidate number 3 is greater than $250,000 
resulting in cance//a*ion of funding (the consequence) is 0.90.” 

Thus, risk and uncertainty are similar, related by their use of probability, but different in that 
risk subsumes probability. The avoidance of conseqences in risk analysis is often introduced in the 
context of comparing two alternatives that have equal expected dollar value. An example is the 
following pair of alternatives: 

Option A: $1000 for sure. 

Option B: A 50-50 chance of zero dollars or of $2000. 

Although both options A and B have equal expected dollar values of $1000, they may not have equal 
expected utilities for some individuals. An individual’s preferences between options A and B reveal his 
attitude toward risk in the range $0 to $2000: 

(1) An individual preferring A to B is characterized as risk-averse. 

(2) An individual preferring B to A is characterized as risk-prone. 
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(3) An individual indifferent between A and B is characterized as risk-neutral. 

In the context of AfcR candidate evaluation, risk is apparent in the following hypothetical 
situation: 

Option C: Crew productivity savings of 240 EVA hours with an initial cost of $1,000,000. 

Option D: A 50-50 chance of saving 180 EVA hours or of 300 EVA hours with an initial cost 

of $1,000,000. 

Both options C and D have equal expected crew productivity savings and equal initial costs but 
individuals may exhibit different preferences as with the previous dollar example. An individual, 
preferring Option C to Option D is characterized as risk-averse. 

Risk attitude implies a certain shape of the individual’s utility function and vice versa (see 
References 5 and 8). A risk-averse attitude for an attribute is equivalent to a concave utility function 
for that attribute. Also, risk-proneness is equivalent to a convex utility function; and finally, risk- 
neutrality is equivalent to a linear utility function. All three of these shapes are illustrated in Figure 3- 
6 for an increasing utility function. An increasing utility function exists for an attribute for which the 
decision maker prefers higher values to lower values. 

The attitude of an individual toward risk varies with the range of outcomes. For example, few of 
us who would prefer Option B above would give up $1,000,000 for sure for a 50-50 chance at zero or 
$2,000,000. Nevertheless, variation in individual attitude toward risk is evidenced by many motorists 
who drive from Los Angeles to Las Vegas to gamble (risk-prone), yet carry collision insurance on their 
automobiles (risk-averse). 

2. Incorporation of Uncertainty in Multiattribute Decision Making 

Uncertainty, as used in the context of the study, is generally incorporated in multi- 
attribute decision making as discrete probability estimates at the chance nodes of the decision tree. 
The more difficult problem is the case where an attribute may take on any (infinite) number of 
attribute states over its range. 

There may be little doubt that a given A&R candidate will cost $250,000 and so a point estimate 
is sufficient. However, the same candidate may save between 100 and 200 EVA hours but it is not at 
all clear what attribute state might actually occur as an outcome due to a host of uncertainties 
stemming from design choices, operations factors, and unforeseen contingencies (to name a few). Such 
uncertainties are better represented with probability distributions. The implementation used in this 
study requires an alternate, but equivalent form of the probability density function, the cumulative 
distribution function or CDF (Reference 31). The probability distribution contains more information 
than the point estimate and it is of interest how the varied mixture of attribute state point estimates 
and probability distributions are combined to derive a probabilistic estimate of multiattribute utility. 
With such a probabilistic estimate, the variance and standard deviation can be used to explicitly view 
the level of aggregate uncertainty due to the combined uncertainties in all of the attribute state 
distributions. 

Thus, the technical manager can be presented with a more powerful representation of preference. 
Rather than presenting only the deterministic multiattribute utilities and forcing the decision maker to 
accept the A&R candidates rank ordered on the assumption of equal uncertainty among the attribute 
states, the mean and variance (or in another form, the standard deviation) of each multiattribute 
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utility can be provided. The relative magnitude of the uncertainties can be viewed explicitly to reveal 
the high-value, low- risk options. For example, comparing a high value, high uncertainty A&R 

candidate with a lower value, low uncertainty AfcR candidate provides a way for the technical 
manager to select lower value options based on minimizing uncertainty and hence, risk— the risk of 
selecting the wrong candidate. 

3. Incorporation of Risk in Multiattribute Decision Making 

Risk has usually been incorporated in multiattribute decision making by taking the 
individual decision maker’s utility functions as probabilities of various outcomes and combining them 
to obtain an expected multiattribute utility for each decision alternative. Alternatives can then be 
ranked in order of expected multiattribute utility with the higher expected utility being the more 
preferred. The incorporation of risk in such a ranking occurs because the individual’s attitude toward 
risk is embodied in the utility functions used to calculate expected utility. If the individual is risk- 
averse, then his multiattribute utility function will yield lower utility values for riskier alternatives. 
Similarly, if the individual is risk-prone, riskier alternatives will have higher utility values. 


U U 



XXX 


(X = Growth Potential) 


Figure 3-6. Examples of Increasing Utility Functions for 
Different Risk Attitudes 
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C. CONCORDANCE 

It is important to determine the extent of agreement among interviewees as to the ranking of the 
alternative AfcR candidates. The analyst is faced with a set of outcomes as in Figure 3-7. Judges I 
and II agree on the most important alternatives (A and C) and judges II and III disagree on the most 
important alternatives. The issue becomes, “Is there any general measure of agreement for the group of 
three judges (or k judges for that matter)?” 

One such measure from the field of rank correlation is a statistic known as Kendall’s Coefficient 
of Concordance (Reference 32). This statistic varies between zero and one, with one corresponding to 
exact agreement among the judges and lower values indicating a greater degree of disagreement. The 
statistic has a known probability distribution and so tests of significance can be performed. 

The hypothesis that the set of rankings produced by a number of judges is independent is tested. 
The null hypothesis, if accepted, would imply disagreement among judges. The more decisively one 
rejects this null hypothesis, the greater is the agreement, or concordance, among the judges. 

Kendall’s Coefficient of Concordance, W, is given by the following equations: 
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and N = Number of alternatives 
k Number of judges 

R.= Sum of the ranks assigned to alternative j 
tjj= Number of tied observations: rank j and judge i 1 


Tables for 5% and 1% significance points are available for selected S values (the unnormalized statistic) 
and various values of k and N (Reference 33). Note that when N > 7 the statistic k (N — 1)W has 
(approximately), a chi-square distribution with N — 1 degrees-of- freedom. Thus, when k (N — 1)W 
exceeds the critical significance point, the null hypothesis of independence of rankings, or lack of 
concordance among the judges is rejected. 


1 The ranks, Rj, of tied observations are taken as equal to the average of the ranks they would have 

been assigned had no ties occurred. For example, suppose Five alternatives, a through e, are 
ranked (from best to worst) d, a, c, e, b, with c and e tied. Ranks would be assigned as 

follows: Ctrl, a-2, c-3.5, e-3.5, b-5. 
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RANKINGS FOR EACH ALTERNATIVE, 

i, BY JUDGE, j 

Alternatives 

I 

Judge 

II 

III 

i. 

A 

2 

2 

i 

2. 

B 

3 

3 

3 

3. 

c 

1 

1 

2 

4. 

D 

4 

5 

5 

5. 

E 

5 

4 

4 


Figure 3-7. Sample Rankings for Concordance Analysis 

Given a set of N alternatives and their rankings— one for each member of the group, it is now 
possible to test the hypothesis for agreement or disagreement for the individuals as a group. As will 
be shown later, this is a straightforward process. The problem still remains that even though the 
individuals may agree as a group, disagreements may exist between judges over specific alternatives. 
Before addressing this problem, a second approach for group decision making is described— computing a 
group ranking from the individual rankings. 

D. ARGUMENTATION ANALYSIS 

The evaluation and ranking of new technologies is hampered by the lack of quantitative attribute 
data. High-leverage prototyping A&R candidates are no exception. Because historical data are rare in 
such cases, the assessment of attribute states is often reduced to educated “guesswork” by individuals. 
Unfortunately, such procedures may involve a mixture of differing assumptions and reasoning for 
estimating the attribute states. Although attempts are sometimes made to capture the rationale for an 
attribute state estimate, the rationale is a weak measure of reliability or grounding because it provides 
only a limited and somewhat unstructured insight as to the basis or reasoning for the estimate. The 
high levels of uncertainty in attribute states for unproven A&R candidates requires a more formalized 
process for understanding the rationale behind each estimate. The following approach used in this 
study attempts to develop an understanding of the patterns of reasoning underlying the rationale. 

Models of attribute state estimates (point estimates or probability estimates) can be viewed as 
sets of propositions or logical statements (arguments) selected to convince the user that the estimates 
are credible (Reference 34). Such arguments represent the basis of the model used to derive the 
estimates. The mode of discourse for situations with very limited or no quantitative information often 
reverts to a measure of validity based on argumentation. A qualitative framework exists for capturing 
the structure of complex arguments based on the the work of Toulmin (References 35 and 36). This 
framework, consisting of subdividing arguments into components, has been used and applied in a 
number of situations involving unstructured and uncertain environments (References 37, 38, and 39). 
Argumentation analysis has also been proposed as a way of understanding the underpinnings of 
attribute state data in multiattribute decision systems (see Reference 39). 
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Within the Toulmin framework, every argument involves a claim (assertion) of truthfulness of a 
given set of conditions or proposed set of actions. The claim of an argument is usually preceded by the 
term, u Then ...” or “Therefore To justify a claim, every argument appeals to a body of evidence 
supporting the claim. Within the Toulmin framework (Figure 3-8), this component is called data. The 
use of data or evidence is typically preceded by the word “Given ...”. The license which enables the 
transition from the data to the claim is called the warrant. The use of the warrant in an argument is 
preceded by the word “Because ...”. The warrant and data constitute the major and minor premises of 
an argument. The backing attempts to justify the relevance of a particular set of data to the support 
of the claim. The backing does not address technical reliability, validity, or objectivity of data (this 
would involve another argument structure for the data collection procedures). The backing is typically 
preceded by the term “Since Finally, the rebuttal represents any or all challenges to any part of 

the main argument— against the claim. The rebuttal is typically preceded by the term “Unless ...”. 

The Toulmin argument framework is applied to identify the argumentation used to support 
claims about attribute state estimates and expose the underlying plausibility of the attribute state 
assertion. A form to elicit not only the attribute state, but the underlying arguments is shown in 
Figure 3-9. The form serves as a basis from which the assumptions, data, and arguments regarding the 
attribute states are used in the multiattribute decision analysis. 

The claim for each attribute state estimate is a numerical response provided by the technical 
assessor. The form of the numerical response can be a point estimate assigned tp the attribute state for 
a particular high-leverage prototyping AfcR candidate (e.g., safety impact = 7.5). Another form of 
the numerical response can be a CDF assigned to the attribute state that reflects the pattern of 
uncertainty in the estimate. Still another form of the numerical response is to simply indicate that the 
attribute state estimate lies somewhere within a range of values but with unknown probability 
distribution (e.g., the attribute state estimate lies somewhere between 6.2 and 8.5). Another 
alternative is the complete lack of sufficient information to characterize even the boundaries of the 
range (e.g., the value lies somewhere between zero and ten but there is no information to indicate 
where). 

The data underlying such claims can be based on the proposed schedule for delivery of products, 
whether any products are to be delivered, the credibility of the funding level required to perform the 
task, and other similar “facts” provided in the task description. The warrants for such arguments can 
be statements such as “the need for the testbed now is based on the objective of delivering the product 
on an early schedule,” or “investment in this task will provide significant operations and maintenance 
cost savings during the life of the project.” The backing reflects a belief in the basic data, for example, 
“as shown in Report X, there is a clear need to perform this task.” The rebuttal simply negates the 
attribute state estimate; for example, “The estimate is too high because there is insufficient time to 
build the testbed prototype,” or “Any operational savings will be offset by the overhead time required 
on the ground which was not included in the analysis.” 

The Toulmin approach is intended to simply identify and organize in a more formal manner the 
process of estimating and justifying attribute state estimates for the alternatives. The aim is to clarify 
the reasoning and background of each numerical estimate that is input to the multiattribute decision 
analysis model. In this manner, not only can the uncertainties inherent to the process be evaluated, 
but the credibility of the results can be placed in context. 
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.SINCE: CURRENT APPROACHES ARE 
LIMITED BY COGNITION AND 
CREW TIME IS VALUABLE RESOURCE 


Figure 3-8. Example Toulmin Argument Structure 


E. APPROACH SUMMARY 

The above approach is summarized by the following stepwise procedure: 

(1) Define the alternative A&R candidates. Simultaneously define the objectives hierarchy, 
decision tree (if applicable), attributes, and attribute scales as required. Identify 
individuals with an interest in the outcome of the decision to serve as decision making 
representatives for the relevant interest groups. Revise attributes with interest groups to 
finalize attribute definitions. Finalize list of A&R candidates. 

(2) Obtain attribute state data in either point estimate or cumulative distribution form. 
CDF’s may be obtained directly from other studies or indirectly using subjective 
probability assessment interviews to elicit the distributions from the participating 
individuals. Simultaneously, at the point the attribute state is defined, obtain the 
Toulmin arguments for and against the attribute state assessment. 

(3) When all of the attribute state data have been specified, conduct utility assessment 
interviews with the interested parties participating in the preference assessment to obtain 
attribute utility functions and trade-off scaling constants. 

(4) Collect and organize data from (3) and (4) and perform multiattribute decision analysis 
computations, sensitivity analysis, and concordance analysis for cases to be studied. 
Report results and conduct additional analysis as required. 


The above steps provide a brief overview of what is, in fact, a detailed and involved process. The 
next section describes an application of the methodology. 












A&R CANDIDATE EVALUATION EXERCISE: HIGH-LEVERAGE PROTOTYPES 
FIRST LEVEL ARGUMENTS IN SUPPORT OF ATTRIBUTE STATES 
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SECTION IV 
AN APPLICATION TO 

HIGH-LEVERAGE PROTOTYPING TECHNOLOGIES 

A. OBJECTIVES FOR ASSESSING HIGH-LEVERAGE A&R CANDIDATES 

The objectives of the evaluation process are to provide to the Associate Program Director (see 
Reference 2): 

(1) A consistent basis for evaluation (with safeguards). 

(2) A mechanism for reaffirming or checking the analysis of the A&R candidate 
advocates. 

(3) An aggregation of input from all concerned parties, based on a consistent 
set of program evaluation attributes. 

(4) An application of a proven tool for the A&R implementation and evaluation process. 

The culmination of objectives (1) through (3), an assessment of high-leverage prototyping candidates, 
is the application under objective (4) described in this section. These objectives can be synthesized 
into a singular overall goal— apply the methodology to ranking the high-leverage prototyping A&R 
candidates. 

The concept of high-leverage prototyping is to draw on existing R&D efforts by identifying 
relatively low cost technologies for station implementation that are on the threshold of technical 
feasibility. Thus by placing small investments in appropriate technologies, the benefits of those 
technologies can be accelerated. The scope (budget) of the high-leverage prototyping program studied 
herein is approximately $3 million. 

B. DEFINITION OF A&R CANDIDATES 

An informal request was distributed in the fall of 1988 via the ex officio leaders of the station 
A&R effort, the leaders of the Advanced Automation Working Group (AAWG), the Robotics Working 
Group (RWG), and the Artificial Intelligence and Expert Systems Technology Working Group 
(AIESTWG). From this request, 18 A&R candidate concepts were received and during the course of 
the following study, the final list grew to 21 AfcR candidates as shown in Table 4-1. The AfcR 
candidates comprise a broad mixture of technology developments, hardware and software modifications 
to existing systems, testbeds, concept studies, issue papers, new applications, and in some cases, 
simply, new ideas to be examined. The range of costs for the AfcR candidates is from $100 thousand 
to $1.35 million. The final list consisted of 10 candidates from the AAWG, 7 from the RWG, and 4 
from the AIESTWG. The AIESTWG submitted fewer concepts because the group had “pre-evaluated” 
their concepts and were submitting the top candidates from their pre-evaluation. 

C. OBJECTIVES, CRITERIA, AND ATTRIBUTES 

The hierarchy of objectives, criteria, and attributes for evaluating and ranking high-leverage 
prototyping AfcR candidates is presented in this section. The objectives hierarchy is a structure that 
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Table 4-1. Alternative High-Leverage Prototyping A&R Candidates 


NAME/IDENTIFIER 

DESCRIPTION 

DIA 

Diagnostic Expert System for electric power and data management 

STD 

Standards and Tools for Expert Systems 

KBS 

Knowledge- based system for fault detection in data management and 
operations management systems 

EXT 

Testbed for extended fault-tolerant testing 

PRO 

Prototype crew scheduler using advanced search methods 

FAU 

Knowledge-based system for electric power subsystem fault prediction 

ABD 

Abductive methodology application/prototype for fault detection 

ACC 

Accelerated power management and distribution testbed development 

ARP 

IV A rack-level robotics 

VDD 

Decision methods application to operations management application 

ROB 

Software upgrades to demonstration of robotic control of assembly 

AUT 

Study of robotics for assembly and fasteners with testbed 

EVA 

Software improvements to EVA Retriever for S.S. Freedom 

POS 

Thermal posture prediction application 

SIM 

Simulation testbed for robotics analysis 

STE 

Integrated mobile platform with stereo vision 

WOR 

World modeling development for S.S. Freedom study 

ADA 

Study of ADA effectiveness for artificial intelligence and 
expert systems development 

MNT 

Study of know ledge- based system development and maintenance tools 

ES/ 

Standards for S.S. Freedom Knowledge-based operations 

KNO 

Standards and infrastructure for knowledge acquisition 


illustrates the linkage and transition from a broad statement of objectives to specific, measurable 
attributes that meet the needs of the decision model used to rank the AfcR candidates (see Figure 3-3). 
Included in the hierarchy are an overall objective or goal, subobjectives, criteria, and attributes. 

The hierarchy should possess a number of desirable properties. The first and most important 
property is the hierarchy should lead to an appropriate ranking of alternatives— one that accurately 
reflects the preferences of the decision makers. The second property of the hierarchy is its clarity and 
ease of use. Ease of use is critical in order for the ranking to be achieved within time and cost 
limitations. Some aspects of this ease of use include: 

(1) Ease of response for those required to provide preferences for the decision model. 

(2) Ease of obtaining attribute data for alternative A&R candidates with regard to the 
attributes. 


( 3 ) 


Ease of carrying out the sensitivity analysis. 


The top level in the hierarchy is an overall statement of the objective for the high-leverage 
prototyping AfcR candidate. The overall objective for the present study is to identify promising high- 
leverage prototyping A&R candidates. 
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The subobjectives represent subjects that influence the achievement of the main objective. The 
subobjectives are chosen to refine the generality of the objective to categorical areas. The suggested 
categories for the subobjectives include mission, operational, safety, economic, and schedule. 

The criteria are derived from requirements and are used to quantify the performance of A&R 
candidates with respect to their objectives and subobjectives. Criteria are the highest level elements in 
the hierarchy that are intended to be quantifiable. For example, cost is a logical candidate for the 
criterion related to the economic subobjective. Thus, under each subobjective there may be one or 
more criteria. 

The high-leverage prototype application criterion for the economic subobjective is to minimize 
S.S. Freedom Program cost impacts. Such impacts are a combination of two factors: initial costs and 
operations costs. The benefits of a high-leverage prototype are accrued if the operations cost savings 
exceed the initial or investment cost for development and implementation. 

The criterion for the mission subobjective is to maximize productivity. The missions performed 
by the S.S. Freedom can be augmented to different degrees by the high-leverage prototypes and the 
purpose of this criterion is to seek to maximize the productivity of the crew under such prototypes. 
Productivity is not simply crew-time per se but rather, implementation of tasks not otherwise 
performed. The productivity is enhanced when the high-leverage prototytpe A&R candidate accelerates 
or replaces a task so the crew is available to perform other important work. This is characterized 
through two elements— productivity enhancement of the crew on-orbit and on the ground. 

The criteria for the operational subobjective are to maximize performance of the S.S. Freedom 
and the capability to accommodate new technology upgrades. Maximizing performance refers to the 
impact of the high-leverage prototype on the overall operational performance of the S.S. Freedom 
which is interpreted here as the effective and efficient utilization of resources to carry out operations. 
Maximizing accommodations refers to the impact of the high-leverage prototype in facilitating the 
evolution of the S.S. Freedom as new technologies are developed in order to improve the productivity 
and performance of the system. 

The criterion for the safety subobjective is to provide improved safety through design features 
(e.g., quick system diagnosis) and reductions in extra-vehicular activity (EVA) of the crew. The 
purpose of this criterion is to account for any contribution of the high-leverage prototype to improved 
safety as a result of its implementation. 

The criterion for the schedule subobjective is to minimize the development time of the high- 
leverage prototype AkR candidate. Although there are many risks associated with building a system 
as complex as the S.S. Freedom , the focus of this criterion is on development time because the costs of 
not meeting schedule milestones can be enormous in comparison with early deliveries. 

Below the criteria are attributes which measure the extent to which each of the criteria are 
satisfied. As described in Section III, the list of attributes should meet a number of requirements in 
order to provide as accurate a representation of each A&R candidate as possible. The attributes 
developed for this study are based on earlier AfcR planning and evaluation activities conducted for the 
S.S. Freedom Program (Reference 40). 

The attributes are described in the next section and illustrated in the objectives hierarchy of 
Figure 4-1. This set includes an overall objective, five subobjectives (economic, mission, operational, 
safety, and schedule) seven criteria, and eight attributes. The eight attributes selected proved to be 
manageable within the scope of the study and after the interviews were completed, no significant 
attribute was found to be missing from the final set chosen. 
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Figure 4-1. Objectives Hierarchy for High-Leverage Prototype Evaluation 
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D. ATTRIBUTE DEFINITIONS AND SCALES 

Because the high-leverage prototyping A&R candidates are being examined in the context of an 
unbuilt system (the S.S. Freedom ), obtaining actual measurements of specific economic and 

operational cost/benefit data is virtually impossible* For this reason, all of the attributes identified in 
the study require estimates of value. Each attribute is characterized with a narrative scale that relates 
numerical value on the scale to specific descriptions of corresponding value. The attribute scales are 
necessary for the decision model to be applied. Each scale requires a unit measure and an upper and 
lower bound. The upper and lower bounds specify the range of the attribute from a worst case value to 
a best case value. The upper and lower bounds for each attribute are determined so that all high- 
leverage prototype AkR candidates considered would fit within these bounds. If an attribute state 
scale value had fallen outside one of these bounds, the utility of that attribute state could not have 
been calculated. For this reason, the interviews with interested parties are carried out after all the 
attribute states have been estimated. The attributes and their ranges are summarized in Table 4-2. 
The attributes are defined below. 

Initial cost impacts are defined as potential cost impacts (savings) on Baseline S.S. Freedom 
development (DDTfcE) costs (the net S.S. Freedom up-front cost change due to funding this high- 
leverage prototype AkR candidate). For most high-leverage A&R candidates, it is sufficient to limit 
S.S. Freedom net cost impacts to the subsystem in which the A&R candidate resides. Operations cost 
is defined as potential operations cost (savings) from the expected outcome of funding the high-leverage 
A&R candidate. Operations cost is estimated at the Station-level if possible, otherwise, it is estimated 
at the level of the subsystem in which the newly automated function resides. The attribute scales for 
intial and operations cost are defined in Tables 4-3 and 4-4. 

Crew productivity is defined in terms of the potential impact of the high-leverage prototype 
A&R candidate toward improvement of Station-wide crew time (productivity) by enabling additional 
productive work. The attribute scale for crew productivity is defined in Table 4-5. 

The attribute of safety represents the potential improvement in safety due to better design or 
reduction in crew hazard exposure. Safety refers generally to (1) a net change in direct crew-time 
involvement in operations judged potentially hazardous directly to the crew member (such as 
EVA), or (2) to the Station at large (such as contingency management operations with little 
margin for response error where a high-speed, high-leverage prototype (e.g., fault diagnosis) could 
improve safety margins). The attribute scale for safety is defined in Table 4-6. 

Development risk represents the likelihood that high-leverage prototyping AfcR technologies 
necessary to automate the function(s) will be available at the required time and within a reasonable 
margin of presently estimated cost. This attribute is analogous to probability of success although 
it is not characterized as a formal probability estimate. The attribute scale for development risk is 
defined in Table 4-7. Note that a higher scale value corresponds to a lower (more preferred) risk. 

Resource requirements impacts, a performance subobjective attribute, measure the potential of 
the product(s) of the high-leverage A&R candidate to mitigate resource constraints such as power 
and upmass. The attribute scale for resource requirements impact is defined in Table 4-8. 

Growth potential represents the net change in the station ability to evolve by accommodating 
later upgrades due to high-leverage prototyping A&R candidate products. The attribute scale for 
growth potential is defined in Table 4-9. 

Spinoff Potential for Terrestrial Applications characterizes the potential for future commercial 
application of technologies that is derived from high-leverage A&R candidate-related technologies. The 
attribute scale for spinoff potential is defined in Table 4-10. 
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Table 4-2. Attributes and Their Ranges 


Attribute 

Worst Case 

Best Case 

Initial Cost Impacts 

Candidate provides minimal S.S. 
Freedom benefits at high cost. 
Significant systems integration costs. 

Funding of this candidate will sub- 
stantially reduce DDT&E and system 
integration costs for Phase 1. 

Very large net initial cost savings. 

Operations Cost Impact 

OAiM savings likely to be negative; 
ground O&M savings likely to be neg.; 
neg. O&M savings on crew training. 

Large expected O&M savings by 
implementing candidate. Savings 
more than A&R alone, cross-cutting 
technology savings are likely. Savings 
extend across lifetime. Large 
O&M savings due to ground opns and 
crew training across lifetime. 

Crew Productivity 

Little or no improvements in crew 
productivity 

Very large improvements in crew 
productivity are likely if the deliver- 
ables are implemented. 

Safety 

No improvements above minimum reqts. 
on orbit or ground. 

Very large improvements are likely if 
deliverables are implemented. Adds 
safety benefits beyond man-in-the-loop 
due to high-speed computing and/or 
reduction in EVA activities. Hazards 
can be mitigated by diagnosis and 
transition to automated "safe-hold." 

Development Risk 

Very high risk development; elements at 
this level not demonstrated; may not be 
feasible. Requires new testbed concepts. 

Virtually no development risk; off-the- 
shelf technologies, primarily mods, to 
current system— software, minimal 
hardware mods.; no testbed facilities; 
low cost. 

Resource Requirements 

Could increase power consumption and 
mass requirements. 

Could result in large power and 
upmass savings. 

Growth Potential 

Proposal would result in some negative 
net impacts on S.S. Freedom ability to 
evolve and accept later upgrades. 

Proposal would result in a large net 
improvement in S.S. Freedom’s abil- 
ity to accept hooks and scars. 

Spinoff Potential 

Little, if no terrestrial applications. All 
developments are S.S. Freedom-specific 
and yield minimal technology transfer. 
Main sources of transfer are documents 
describing developments and techniques. 
Few generalized methods or approaches. 

Proposal would have very high level of 
application to medium and low tech, 
areas. Technologies will permeate 
into large markets (e.g., households). 
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Table 4-3. Attribute Scale for Initial Cost Impacts 


Scale 

Val ue Rating Attribute Description 

Best Case 10 Funding of this proposal will substantially reduce DDTfcE and system 

integration costs for the Phase I SSF. Very large net initial cost savings. 

8 This proposal will reduce systems integration costs and potentially 

reduce DDT&E costs for the Phase I SSF. Initial cost savings from the 
investment are very likely. 

6 This proposal will have minimal DDT&E cost savings for the Phase I 

SSF. Impacts are likely during the operations/evolution phase. 

4 Proposal supports SSF assembly but DDTfcE costs could be significant. 

2 Proposal supports SSF assembly but DDTfcE costs could be large. 

Some issues for systems integration. 

Worst Case 0 Proposal provides minimal SSF benefits at high cost. Significant 

system integration costs. 
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Table 4-4. Attribute Scale for Operations Cost Impacts 


Value and On-Orbit Costs 

Scale Rating Description 

Ground Operations 
Costs Description 

Crew Training 
Costs Description 

Best 

Case 

10 

Large expected O&M savings due 
to implementation of proposal 
deliverables. Savings are more 
than A&R alone; cross-cutting 
technology savings are likely. 
Savings extend across SSF 
lifetime 

Large O&M savings due to 
proposal are robust— savings 
impacts will be obtained for grnd 
ops. across SSF lifetime. 

Large O&M savings on 
costs due to implementation 
of proposal deliverables are 
likely. 

7 

O&M savings are focused with- 
in specific subsystems. Savings 
extend across SSF lifetime. 

Significant grnd. O&M savings 
across SSF lifetime sufficient 
to break even against DDT&E 
cost. 

Significant O&M savings on 
training costs. 

5 

O&M savings are likely and 
primary savings are expected 
during the 1st half of the SSF 
lifetime. 

Grnd. O&M savings are 
sufficient to break even when 
weighed against initial invest- 
ment cost (DDT&E). Savings 
will occur during 1st half of 
SSF lifetime. 

Some O&M savings in crew 
training costs. 

3 

O&M savings are expected 
during the 2nd half of the SSF 
lifetime. 

Grnd. O&M savings are 
uncertain and likely to occur 
during the 2nd half of the SSF 
lifetime. 

The crew training benefits 
for the proposal are 
uncertain. 

0 

Worst 

Case 

O&M savings are likely to be 
negative (no savings). 

Grnd. O&M savings are likely 
to be negative. 

Negative O&M savings on 
crew training costs. 
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Table 4-5. Attribute Scale for Crew Productivity Impacts 


Value 

Scale 

Rating 

On-Orbit Productivity 
Description 

Ground Productivity 
Description 

Best Case 

10 

Very large improvements in crew 
productivity are likely if the proposal 
deliverables are implemented. 

Large improvements in ground crew 
productivity are likely. 


7 

Significant Improvements in crew 
productivity are likely. 

Significant improvements in ground 
crew productivity are likely. 


5 

Some improvements In crew 
productivity are likely. 

Some improvements in ground crew 
productivity are likely. 


3 

Few improvements in crew 
productivity are likely if the 
proposal deliverables are 
implemented. 

Few improvements in ground crew 
productivity are likely. 

Worst Case 

0 

Little if no improvements in crew 
productivity are likely. 

No improvements in ground crew 
productivity. 
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Table 4-6. Attribute Scale for Safety 


Value and On-Orbit Safety Ground Safety 

Scale Rating Description Description 


Best Case 
10 

Very large improvements in crew safety 
are likely if the proposal deliverables are 
implemented. Adds safety benefits be- 
yond those achievable with man-in-the- 
loop due to high speed computing and/or 
reduction in EVA. Hazards can be 
mitigated by diagnosis and transition of 
system to automated "safe-hold" mode. 

Large improvements in ground safety are 
likely. 

8 

Moderate safety improvement is likely. 
Proposal technologies will increase safety 
via moderate reductions in required EVA. 

Moderate improvements in ground safety are 
likely. 

6 

Some improvements in safety. 
Implementation of proposal deliverables 
likely to increase safety by preventing 
improper or ill-defined command sequences 
from entering the control system. No major 
effects on EVA activity. 

Some Improvements in ground safety are 
likely. 

4 

Minor improvement in safety. Addition of 
proposal deliverables increases safety in 
specific areas by monitoring critical 
functions and data reduction of SSF status 
to smaller numbers of parameters. No 
expected effects on EVA. 

Minor improvements in ground safety are 
likely. 

2 

Little improvement in safety. Addition of 
proposal technology may be for reasons 
other than safety but benefits may be 
accrued to safety in specific (and perhaps 
unlikely circumstances). There may be a 
small increase in required EVA. 

Little, if any improvement in ground safety is 
likely. 

0 

Worst Case 

No improvements in safety above minimum 
requirements. 

No improvements in ground safety above 
minimum requirements. 
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Table 4-7. Attribute Scale for Development Risk 


Scale 

Value Rating Attribute Description 


Best Case 

10 

Virtually no development risk; off-the-shelf technologies; 
primarily modifications to the current system— mainly software; 
minimal hardware modifications; no testbed facilities required; 
low cost. 

Some development risk; testing required for minor interfaces 
and verification purposes; existing testbeds can be modified 
and used; software and hardware modifications required; 
demonstrated feasibility; low level cost uncertainties. 


7 


5 

Moderate risk development; some elements may require 
experiments and ground testing; some new testbed facilities 
are required; demonstrated feasibility; moderate cost. 


3 

High-risk development; numerous elements require ground 
testing; new testbeds need to be constructed; some in-space 
testing; feasibility relatively certain. 

Worst Case 

0 

Very high-risk development; elements at this level not demon- 
strated; may not be feasible. Requires new testbed concepts 
be constructed. 
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Table 4-8. Attribute Scale for Resource Requirements Impacts 


Value and Non-Consumables Consumables 

Scale Rating Power Upmass Upmass 


Best 

Case 

10 

The proposal implementation 
could result in large power 
savings. 

Large savings in non-con- 
sumables is likely. 

Large savings in consum- 
ables is likely. 

8 

The proposal could result in 
significant power savings. 

Moderate savings in non-con- 
sumables is likely. 

Moderate savings in con- 
sumables is likely. 

6 

The proposal could result in 
some power savings. 

Some savings in non-con- 
sumables is likely. 

Some savings in con- 
sumables is likely. 

4 

The proposal could 

yield minor savings in power. 

Minor savings in non-con- 
sumables is possible. 

Minor savings in con- 
sumables is possible. 

2 

The proposal would have 
little impact on power reqts. 

Little, if any savings in non- 
consumables. 

Little, if any savings in 
consumables required. 

0 

Worst 

Case 

The proposal could increase 
power consumption. 

Could increase non-consumable 
upmass requirements. 

Could increase consumables 
upmass requirements. 
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Table 4-9. Attribute Scale for Growth Potential 


Scale 

Value Rating 

Best Case 10 

7 

5 

3 

Worst Case 0 


Attribut e Description 

Proposal would result in a large net improvement in SSF’s 
ability to evolve or accept hooks and scars. 

Proposal would result in a moderate net improvement in SSF’s 
ability to evolve or accept later upgrades. 

Proposal would result in no net change in SSF’s ability to 
evolve or accept later upgrades. 

Proposal would result in minor negative net impacts on SSF’s 
ability to evolve and accept later upgrades. 

Proposal would result in some negative net impacts on SSF’s 
ability to evolve and accept later upgrades. 
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Table 4-10. Attribute Scale for Spinoff Potential 


Value 

Scale 

Rating 

Attribute Description 

Best Case 

10 

Proposal would have very high level of application to medium 
and low technology areas. Technologies will permeate into 
large markets (e.g., households). 


7 

Proposal would have many applications. Automation activities 
are applicable to medium technology areas and usable for many 
areas of manufacturing and production. 


5 

Proposal would yield some terrestrial applications. Main 
terrestrial examples are spinoffs to high technology applications 
such as computer software and manufacturing. 


3 

Few terrestrial applications. Proposal developments are aimed 
at aerospace and military applications where costs are high 
and technology-specific (e.g., automated navigation and attitude 
control). Main terrestrial applications are spinoffs from SSF 
to other space applications such as unmanned satellites. 

Worst Case 

0 

Little, if no terrestrial applications. All developments are 
SSF-specific and yield minimal technology transfer. Primary 
sources for transfer are documents describing developments 
and techniques. Few generalized methods or approaches to 
automation and robotics. 
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E. A&R CANDIDATE ATTRIBUTE STATE DATA 

In general, the most difficult step in any evaluation is the collection of data about the alternatives 
being evaluated-this study was no exception. A variety of data collection procedures were examined 
in response to critique from the users the evaluation was originally designed to serve— the working 
group chairmen of the AAWG, RWG, and AIESTWG. The initial approach was to charge the 
working group leaders with the coordination function of data collection. The list of attributes and 
scales was to be distributed via the working groups to the authors of the high-leverage prototype A&R 
candidates who would respond formally through their respective working group. 

1. Objections to the Process and Safeguards 

Unfortunately, the entire process became embroiled in debate which, coupled with limited 
working group resources and time resulted in a stalled situation. Although a formal agreement had 
been made by all involved, at the point where the attributes were being finalized and the collection of 
attribute data was to begin, a variety of questions was raised to challenge the fairness of the process 
and its vulnerability to manipulation of input data. The following comment characterizes the 
argument: 

" Prior to rating any particular proposal, scale factors must be determined for each attribute. This 
determination is human-performed and is subject to individual bias and parochial interest. It is a well 
known fact that mathematics can be creatively utilized to prove just about anything . Therefore , if one 
were to know the equation applied, one would be able to produce any parochial result desired. Thus, 
the evaluation process can be easily short-circuited and defeated. Even without specific knowledge of 
the mechanics of the equation, most of us are well enough imbued with the process to create desired 
results. ... In short, I think that there are important flaws in the strawman, both as currently 
configured, and with its use as a decision-making aide. I hope this is the constructive input to your 
activity which you are seeking (Reference 41)." 

This comment raises two issues. The first issue is the problem of bias or gaming the inputs in 
such a manner as to gain advantage in the ratings. A number of safeguards minimize the potential for 
such effects. 

The first, and most important safeguard is the performance of the evaluation process by a third 
party with no stake in the outcome of the selections. 

The second safeguard is the assignment of attribute ratings by a third party. Allowing the 
proposers to assign the attribute data may have been the primary concern here. An additional, related 
safeguard is the development of arguments and counterarguments to support the attribute state data 
assignments. The ability to include uncertainty addresses concerns about the subjective nature of the 
estimates to a degree. If the uncertainties are large, the analysis should reveal and quantify this. 

The third safeguard is the separation of the attribute rating and attribute prioritization processes, 
A different group of individuals assigned the attribute state values from the group of interviewees who 
provided the attribute utility functions and trade-off scaling constants. In addition, the interviewees 
providing the priorities (weighting factors) were not presented with any information that would allow 
them to associate any attribute states with specific high-leverage prototytpe A&R candidates. 

The fourth safeguard is the involvement of multiple decision makers across different 
organizational groups to help assure a balanced view of the process. 

The second issue raised by the above comment is the relevance of the attributes to the high- 
leverage prototype A&R candidates under consideration. It is not clear the list of attributes in some 
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way penalizes different classes of high-leverage prototype A&R candidates because hooks and scars 
candidates should show benefits under growth potential, while the design tools, standards, and 
architecture candidates should show benefits under a number of attributes such as initial cost, 
development risk, and spinoff potential. 

As these issues were addressed, it became increasingly apparent that involving the working group 
leaders to provide attribute state ratings was not workable from their positions as Level II agents and 
as advocates for high-leverage prototype A&R candidates within their respective working groups. 
Thus, it became clear the attribute state data could not be expected from (1) the authors of the high- 
leverage prototype A&;R candidates because of concerns over biasing of inputs, (2) the working group 
chairmen because of their representation of their working groups, or (3) the systems engineering and 
integration function due to limited resources for A&R support. The next section describes a technical 
assessment group formed to support the assignment of attribute state values. 

2. Technical Assessment Committee 

To achieve the objectives of the study, a third-party group was formed to assess the 
attribute states for each high-leverage prototype AfeR candidate. Five individuals participated from 
the Jet Propulsion Laboratory staff— an organization with the ability to understand the technology 
alternatives, but no direct interest in the selection of any high-leverage prototyping AfcR candidates. 
Two meetings were held to (1) establish a set of ground rules for the assessment, and (2) review the 
ground rules after performing a first round exercise to estimate the attribute states to identify any 
common problems. 

3. Ground Rules and Procedure 

A major difficulty facing the technical assessment committee was the lack of detailed data 
on not only the attribute states, but the high-leverage prototype A&R candidate itself. The variation 
in quality of information varied from extensive 15-page descriptions to one-page summaries. 
Fortunately, a number of the technical assessment committee members were involved in working group 
meetings where the authors of the high-leverage prototype A&R candidates described each concept in 
greater detail. No contacts were made with these authors in order to maintain fairness and reduce bias 
to the extent possible. 

At the first meeting of the technical assessment committee, a compilation of the latest data for 
each high-leverage prototype A&R candidate was distributed to each member. A discussion of the 
requirements and purpose of the group was followed by the development of a set of ground rules to be 
used as guidelines for estimating the attribute states. It was recognized by all that high levels of 
uncertainty would be systemic to the process and the guidelines were established to facilitate what was 
viewed as a difficult, but not futile task. The ground rules were: 

(1) Any reference to the S.S. Freedom is assumed to include space operations and ground 
operations. 

(2) The high-leverage prototype A<kR candidate must contribute in a high-leverage context. 

(i.e., the concept must perform an enabling or a support function (less desirable)). 

(3) The high-leverage prototype A&R candidate must have a demonstrable product. There 
must be specific hardware, software, concepts, methodology, application, or a study of 
issues delivered. 
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(4) There should be sufficient information for evaluation and extrapolation 
of attribute states. 

(5) Consideration will be given to the latest, most detailed information. 

(6) Give the benefit of doubt to deliverables promised and schedule 
proposed unless the supporting arguments for these are weak or it is 
known otherwise that they are unachievable. 

(7) The growth potential attribute ratings are aimed at S.S. Freedom accommodations as 
opposed to growth of the high-leverage prototype AfcR candidate. 

(8) It is assumed that any successor tasks described would be funded through to completion. 

After the ground rules were established, subsets of the attributes were assigned to each member of 
the group. Thus, each individual was responsible for all the attribute states for two attributes across 
all the high-leverage prototype A&R candidates rather than assessment of all the attributes for a 
selected number of candidates. Because of large uncertainties in the study, estimates of attribute states 
could take a number of forms from a total lack of knowledge about the attribute value to a “best guess” 
estimate. 

The most uncertain form of attribute state can be represented with a uniform probability 
distribution over the attribute range. The uniform distribution, in effect, allows the attribute state to 
take on any value over the attribute range with equal probability. Because the attribute range is frpm 
zero to ten, such a distribution simply indicates the “true” value of the attribute state is somewhere 
between zero and ten and no more. 

Another form of the same distribution which adds some additional information is the uniform 
distribution over a smaller range than zero to ten. For example, if an attribute state is uniformly 
distributed between two and four, although it could be anywhere between two and four with equal 
probability, the range has been reduced from 0-10 to 2-4 which narrows uncertainty about the location 
of the true estimate to a more specific region of the scale. 

A third form of distribution which again narrows the location of the attribute state estimate is 
the assignment of a non-uniform probability distribution. Within this study, such distributions are 
obtained using subjective probability assessment interviews to elicit the probability distribution. 

Finally, if the attribute state estimates are known with certainty or believed to be distributed 
with equal uncertainty across all the estimates, the average or mean value can be used. These values 
(often called “best guess” estimates) are point estimates that provide no quantifiable information about 
underlying uncertainties. 

The implications of the above forms were explained to the technical assessment committee and a 
series of blank attribute state forms were distributed (see Figure 3-9). The task was clear— the attribute 
state estimates were needed for 168 attribute states (eight attributes for each of twenty-one high- 
leverage prototype AfcR candidates). Each attribute state (distribution or point estimate) was 
estimated and the argument(s) for and against the estimate were documented to provide the reasoning 
process. 


1 
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4. Sample Questions 

The technical assessors were given blank forms on which to provide the attribute state 
estimates in a number of forms (see Figure 3-9): 

(1) The point estimate can be provided directly. 

(2) The cumulative distribution function can be specified numerically by providing the 0, 25th, 
50th, 75th, and 100th percentiles. 

(3) The cumulative distribution function can be specified graphically by sketching the form of 
the function. 

(4) The cumulative distribution function can be elicited using subjective probability 
assessment techniques to obtain an estimate of the technical assessors uncertainty 
surrounding the attribute state estimate. 

Figure 4-2 illustrates an example of one of these completed forms. At the completion of this 
activity, the attribute state data were reviewed by the committee. The results are reported in the next 
section. 


5. Attribute State Data 

The final database of 168 attribute states is summarized in Figure 4-3 showing the distribution 
values as a mean and standard deviation and the point estimate values as a single number. Sixty-nine 
percent (69%) of the 168 attribute states are characterized by cumulative distribution functions (116 
attribute states) and thirty-one percent (31%) of the attribute states are point estimates (52 attribute 
states). 

An indication of the level of uncertainty is provided by the fact that fifty-seven percent (57%) of 
the attribute states are represented by some form of uniform cumulative distribution function (66 
attribute states). Five percent (5%) of the attribute states were full-scale uniform distributions where 
no further judgment could be made regarding resource requirements impacts, growth, and spinoff 
potential (6 attribute states). The next section describes how the attribute states were used in the 
interview process to obtain the utility functions and trade-off constants from the interested parties. 

F. INTERVIEWS TO OBTAIN PREFERENCES FROM INTERESTED PARTIES 

The purpose of the interviews is to obtain the quantified preferences of the interested parties in 
order to rank alternative high-leverage prototype A&R candidates with the decision model. Equally 
important is the provision of a formalized process to enable different stakeholders in the decision 
process to provide input about their preferences. The preference information required from each 
interested party includes a utility function and a trade-off scaling constant for each attribute. The 
utility functions characterize how the interested party prefers different levels of an attribute while the 
trade-off scaling constants characterize how the interested party values one attribute versus the other 
attributes. Interviewees were sought who were knowledgeable about automation and robotics issues as 
they pertain to the Space Station Program, and, who were regarded as decision makers within their 
organizations. 
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1. Interviewees 

The interviewees were persons who would either have a direct role in the development of 
the high-leverage prototype A&R candidates or who act as advisors in the decision-making process. 
Representatives were sought from three organizations: 

(1) NASA Headquarters (Level I) representatives with a NASA programmatic perspective. 

(2) NASA Space Station Program (Level II) representatives with a Space Station Program 
perspective. 

(3) NASA Third Party Technology Program representatives with an A&R technology 
perspective. 


A total of nine individuals were interviewed composed of three from each group between April 17, 
1989 and April 28, 1989. Three of the interviewees are technology program managers, two are working 
group leaders, three are AfcR technologists, and one was directly involved in the S.S. Freedom 
operations area. While other groups might have been interviewed (such as contractors or user payload 
representatives), this group comprises a significant cross-section of interested parties concerned with the 
high-leverage prototytpe AfcR candidates. 

The representation of members in the sample was constituted from an initial survey of 
representatives derived from meeting agendas, personal contacts, and referrals. While this sampling 
approach is not a random one, there were a number of individuals who simply had to be included 
because they had played a key role in some aspect of advanced A&R technology development. Using a 
randomized sample and possibly omitting them from the survey would have left serious gaps in the 
results of the study. Furthermore, a larger, random sample would tend to move the results toward 
some “average” set of responses. The aim of the interviews was to survey those at the leading edge of 
advanced A&R technology development and obtain an informed, critical response as opposed to an 
average or typical response. Although more interviews might have been desirable, the time and 
resources to accomplish them were not available. 

2. Interview Process 

The selected personnel were asked to provide their inputs to the rankings during one-hour 
interviews although the interviews ranged from 50 to 90 minutes. These sessions were highly 
structured to obtain the interviewee’s utility functions and trade-off scaling constants for the set of 
eight attributes. 

The interviews were divided into six steps (Figure 4-4). The First step involves an introduction to 
the study and its purpose with a presentation of the attributes and their scales and ranges. The second 
step elicits the utility functions for each attribute. The third step checks for independence conditions 
that are important for methodological reasons by asking if the responses to the questions of step 2 
would vary with changes in the levels of the other attributes (attributes other than the attribute for 
which the utility function is being assessed). If the interviewee does believe the attributes are 
dependent, an additional set of questions are used to determine the conditional utility functions. The 
fourth step involves ranking the attributes in order of their importance. This step is used as a check 
for preference reversals in the fifth step where the trade-off scaling constants are obtained, to insure 
that responses are consistent. The sixth step is a summary of the interview process and provides the 
interviewee with an opportunity to make additional comments and revisions as needed. 







Figure 4-4. Interview Process 
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3. Sample Questions 

Sample questions used during the interviews are illustrated by Figures 4-5 (step 2, utility 
assessment), 4-6 (step 4, ranking of attributes), and 4-7 (step 5, trade-ofT scaling constant assessment). 
Figure 4-5 presents a sample question used to obtain the information needed to construct the 
interviewee’s utility function for the attribute “growth potential. 75 Figure 4-6 presents a sample 
question to obtain the interviewee’s ranking of attributes by importance. Figure 4-7 shows a sample 
question for obtaining the scaling constant for the attribute “growth potential. 75 The full questionnaire 
is contained in Appendix A of this report. 

The use of interviews in the decision analysis process is well established and documented 
(References 42, 43, 44, and 45). The method used in the present study for utility function assessment 
is called the certainty-equivalent method because it involves a search for an attribute indifference point 
that is “certain” versus an equivalent lottery with fixed probabilities (50-50). Recent studies have 
focused on limitations of the certainty-equivalent process used and a number of alternative 
questionnaire techniques have been under development such as the lottery-equivalent method where the 
indifference choice is between two lotteries rather than a certainty versus a lottery (References 46 and 
47). Comparison studies have not yet demonstrated improved performance of these alternate methods 
in an applied setting and such a comparison was beyond the scope of this study. 

4. Interview Results 

For the most part, the interviews went rather smoothly. All interviewees were able to 
provide the information needed to form their attribute utility functions and trade-off scaling constants. 
Table 4-11 summarizes the utility function assessment responses in two ways. The median certainty 
equivalents approximate the characteristics of a preferred high-leverage prototype AfcR candidate. The 
attitudes toward risk for each of the attributes implied by the responses show that the group as a whole 
was risk averse to high operations cost impacts, crew productivity, and resource requirements impacts 
(perhaps realizing that a major benefit of AfcR is generally realized during the operations phase 
through improved productivity and reductions in resources required). The interviewees are 
predominantly risk neutral for safety and development risk (perhaps assuming that minimum safety 
requirements will be met regardless of the high-leverage prototypes and development risk can be 
leveraged with additional funding). The interviewees were somewhat risk averse to initial cost 
attribute (high initial cost for an A&R candidate reduces cost-effectiveness) and definitely risk prone to 
the growth attribute (perhaps by definition, the purpose of a high-leverage prototype is 
implementation— not growth and there is a willingness to gamble for high values of growth as exhibited 
here). Another interesting case is the split in risk attitudes for spinoff potential— an equal number of 
risk averse, risk neutral, and risk prone responses indicating very different assumptions about the 
importance of spinoff potential. 

Responses to the questions for ranking the importance of each of the attributes are summarized 
by group and for the entire sample in Table 4-12. The table presents the overall ranking of the 
attributes on the left and the breakdown by individual ranking in the three columns on the right. Note 
that these are strict ordinal rankings and the reader should not be mislead by the range of rankings. 
For example, in some cases the interviewee ranked two attributes equally resulting in less than eight 
numbers. The relevance of attribute importance is to indicate those factors which tend to influence the 
final rankings. The term “influence” is used because although two attributes, A and B, might be rated 
1 and 4, the actual weights used in the calculations might be 0.90 and 0.88, respectively (i.e., non- 
linear). The conclusion to be drawn is that A is preferred to B— not that A is four times as important 
as B. 
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Figure 4-5. Sample Utility Function Questions 






Figure 4-6. Sample Ranking of Attribute Importance Question 
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Figure 4-7. Sample Trade-off Constant Question 
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Table 4-11. Interview Data for Utility Responses 

Attribute 

Response 

Range 

Median 

Response 

Risk 

Averse 

Risk 

Neutral 

Risk 

Prone 

Initial Cost Impacts 

4.0 - 7.0 

5.0 

2 

3 

4 

Operations Cost Impacts 

2.5 - 7.5 

3.5 

0 

0 

2 

Crew Productivity 

2.5 - 8.0 

5.0 

□ 

3 

2 

Safety 

3.5 - 8.0 

5.0 

3 

□ 

2 

Development Risk 

4.0 - 8.0 

5.0 

2 

0 

2 

Resource Requirements 

2.5- 6.5 

4.5 

5 

3 

1 

Growth Potential 

5.0 - 7.0 

6.0 

0 

3 

0 

Spinoff Potential 

2.0 - 9.5 

5.0 

3 

3 

3 I 


Table 4-12. Interview Data for Importance of Attributes 

RANK SUM RULE RANKINGS INDIVIDUAL RANKINGS BY GROUP 


Attribute 

Rank Sum Rule 
Over All Groups 

Level I 
Group 

Level II 
Group 

Third 

Party 


Crew Productivity 

1 

1,2,4 

1,1,2 

2,3,4 


Operations Cost Impacts 

2 

1,2,2 

1,3,4 

2,2,6 


Initial Cost Impacts 

3 

2,5,6 

4,5,5 

1,1,1 


Safety 

4 

1,4,6 

2,2,4 

3,4,7 


Growth Potential 

5 

3,3,7 

3,4,7 

3,5,7 


Development Risk 

6 

3,4,5 

4,6,6 

3,4,8 


Resource Requirements 

7 

4,6,7 

3,4,7 

3,5,5 


Spinoff Potential 

8 

5,8,8 

4,8,8 

6,6,8 







1 
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The NASA group rankings (Levels I and II) reveal disparities over the priority of crew 
productivity versus operations cost impacts perhaps due to their common importance for operations 
benefits. The Third Party group disagrees completely preferring initial cost (perhaps due to the 
realization that many funding decisions are based on initial costs rather than operations or combined 
life-cycle costs). The groups tend to agree that spinoff potential is not a major factor that influences 
the selection of high-leverage prototyping A&R candidates. The rank sum rule described in Section III 
is used to collapse the individual rankings into the list on the left. Again, this ranking is ad hoc in 
nature and used here to simply show the general importance exhibited by the individual ranks. 

An additional display of attribute importance for the most preferred and least preferred attributes 
is shown in Table 4-13. The apparent importance of crew productivity, operations cost, and initial cost 
is highlighted as is the apparent low importance of spinoff potential. 


Table 4-13. Ranking of Attributes for Most and Least Important Attributes 

Number of Times Rated : 


Most Least 

Attribute Important Important 

Crew Productivity 

4 

0 

Operations Cost Impacts 

6 

0 

Initial Cost Impacts 

5 

0 

Safety 

2 

1 

Growth Potential 

1 

3 

Development Risk 

1 

2 

Resource Requirements 

0 

1 

Spinoff Potential 

1 

UJ 


G. POST-PROCESS EVALUATION 

The final step in the evaluation was to ascertain from all the participants a perception as to the 
strengths and weaknesses of the overall study. The post- process questionnaire is shown in Appendix D. 
The purpose of the post-process evaluation was to obtain an overall reaction of the participants to 
various elements of the study, identify areas that need improvement, and use the critique provided to 
determine whether the process is workable as now constituted or needs modification. 

The cover letter and questionnaire were distributed to twenty individuals and a follow-up letter 
was sent approximately four weeks later. A total of six responses was returned (30%). The results of 
the post-process questionnaire are discussed in Section VI. 


i 
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SECTION V 

MULTIATTRIBUTE DECISION ANALYSIS 
MODEL RESULTS 


A. OVERVIEW 

This section describes the analysis performed using the data sets described in Section IV. The 
alternative high-leverage prototyping AfcR candidates are characterized by the values of their attribute 
states (either point estimates or cumulative probabilities) obtained from the technical assessment 
committee. The preferences of the interested parties are characterized by their utility functions and 
trade-off scaling constants obtained during the interviews. The two datasets are combined using the 
multiattribute decision analysis methodology described in Section III to compute a ranking of the 
alternative high-leverage prototype A&R candidates. A sensitivity analysis is performed to better 
understand the interplay of the numerous parameters and their effects on the results. The focus of this 
section is a description of ranking results for the nominal case which forms the baseline result for the 
study. The nominal case is used as the primary basis for selecting promising high-leverage prototype 
A&R candidates (subject to further analysis). However, a sensitivity analysis of variations on decision 
model assumptions, and an analysis of the implications for the set of high-leverage prototype A&R 
candidates are described via a series of analysis cases, 

1. Analysis Cases 

Four important decision model assumptions are varied in order to test the sensitivity of 
the model and bound the results. The following discussion describes each of the decision model 
assumptions varied, the cause(s) of variation in the results (differences in ranking results), and the 
default decision model assumption used for the nominal case. The four decision model assumptions 
varied are (1) probabilistic versus deterministic analysis; (2) nominal versus best and worst case utility 
functions; (3) multiplicative versus additive decision models; and (4) individual preferences by 
interested party groups. 

The first decision model assumption examined is a comparison of the rankings for the 
deterministic and probabilistic cases. The purpose is to identify differences between rankings in both 
cases to determine the impacts of including uncertainty, and to trace the sources of uncertainty. Any 
differences in ranking results are due to the use of attribute state cumulative distributions rather than 
single-valued estimates. The nominal case uses the probabilistic results. 

The second decision model assumption examined is a comparison of the results implied by the 
nominal data and the best and worst case values obtained in the preference assessment interviews. The 
best and worst case values provide upper and lower boundaries for the range of ranking results as a 
function of dependencies between preferred attributes. Any differences between ranking results would 
be due to the interested party’s belief that dependencies exist between attribute states. The nominal 
case uses the values under assumed independence. 

The third decision model assumption examined is a comparison of the ranking results using two 
different decision models. As described in Section III and used for the nominal case, if the trade-off 
scaling constants do not sum to 1.0, the multiplicative model (Equations 1 and 3) is used to calculate 
the outcome multiattribute expected utility. However, the additive model results (which assume 
attribute preferential independence) can also be computed by normalizing the trade-off scaling 
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constants to sum to one. Any differences between ranking results under this decision model assumption 
would be due to exclusion of the non-linear cross-terms in the multiplicative model. 

The fourth decision model assumption examined is a comparison of the ranking results between 
groups and across groups. Differences are examined between the set of individual rankings and the 
rankings of subsets of individuals. There are four groups in this study: Level I, Level II, Third Party 
Technologists, and a combined group of all the interested parties. The nominal case is the combined 
group of nine interested parties. 

A number of additional cases could have been constructed but were not considered within the 
scope and schedule of the study. 


2. Computer Runs 


The sensitivity analysis of the above decision model assumptions implies the calculation of 
forty-eight different cases: 


Number of Number of 

Case Combinations Input Files 


Probabilistic versus deterministic 

2 

1 

Nominal versus best versus worst case 

3 

3 

Multiplicative versus additive decision model 

2 

1 

Groups 1,2,3, 4 

* 4 

+ 4 

TOTAL NUMBER 

48 

9 


Appendix C lists the combinations of cases and output for the nominal case. A series of five-hundred 
and one-hundred Monte Carlo trial runs were performed in the probabilistic cases, and after 
determining convergence within one-hundred Monte Carlo trials, subsequent probabilistic computer 
runs were made at the one-hundred trial level. The output from the series of runs was printed and 
analyzed. The results of the analysis are presented in the following sections. 

B. RESULTS FOR THE NOMINAL CASE ATTRIBUTE DATA 

The computed rankings for the nominal case provide a list of rank-ordered, high-leverage 
prototype A&;R candidates for each interested party. The analysis procedure consists of displays 
showing the range of rankings across the various groups summarized with a measure of group tendency 
(the median). The analysis discusses general observations and then focuses on the top ten high-leverage 
prototype A&R candidates and the results of varying a number of decision model assumptions. A 
comparison of the results is made to separate evaluations conducted by AfcR working groups and an 
overall summary of results is presented. 

The overall rankings across interested parties are summarized in Figure 5-1 and the separate 
group I, II, and III rankings are presented in Figures 5-2, 5-3, and 5-4 (see Table 4-1 for the definitions 
of the high-leverage prototypes). Each high-leverage prototype A&R candidate is listed on the x-axis 
and the integer rank order (1,2,3, ...) is plotted on the y-axis (note that a ranking of 21st is most 
preferred). The figure is derived in two steps. The first step involves subtracting one standard 
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Figure 5-3. Probabilistic Rankings, Level II Group 


HIGH LEVERAGE PROTOTYPE EVALUATION 
GROUP 111 RANKINGS WITH UNCERTAINTY 
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STD ADA AuT FAU PRO ES/ POS KNO ARP VDD 

CANDIDATE 


Figure 5-4. Probabilistic Rankings, Third Party Group 
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deviation from each outcome utility mean and rank ordering the high leverage prototype AfcR 
candidates. The second step involves adding one standard deviation to each outcome utility mean and 
rank ordering the high-leverage prototytpe AfcR candidates. The range of results is then plotted on 
Figure 5-1 as a range of low and high across all nine interested parties. Thus, the range displays the 
overall position of each high-leverage prototype AfcR candidate relative and the variation due to a 
combination of attribute state uncertainties and interested party preferences. 

Figure 5-1 highlights an immediate problem of how to interpret the results. A simplistic 
approach entails ranking the alternatives based on the median of the nine rankings for each high- 
leverage prototype AfcR candidate. However, the variation associated with each candidate provides a 
measure of the associated uncertainty. The question ultimately becomes, “Should the high-leverage 
prototype AkR candidates be rank ordered based on value only (e.g., the median) or on value plus 
uncertainty?” This question points to a common trade-off facing technical management-whether to 
select a high value, highly uncertain option, or select a lower value, low uncertainty option. Rather 
than select one approach, both are presented here. 

Figure 5-1 does show, in a general sense, the high-leverage prototype AkR candidates in 
descending order of preference. The DIA, STD, and KBS options are most preferred while ARP, 
WOR, and VDD are least preferred. Although the range of variation appears to diminish with value, a 
number of high-leverage prototype AfcR candidates with very wide ranges are interspersed in the list, 
namely ROB, MNT, PRO, and KNO. Some of this uncertainty can be traced to the Level II group for 
the cases of PRO and KNO. The broad range is due to a difference of opinion in the Level II group. 
This can also be seen by comparing Level I (Figure 5-2) and the third party group (Figure 5-4) to the 
Level II group (Figure 5-3). Note that Level I and the third party group have much smaller ranges 
and in some cases no range (indicating the members agreed on that high-leverage prototype AfcR 
candidate rank), while the Level II group has a much wider range. In fact, one individual representing 
an operations perspective in the Level II group widens the range not only for that group but for the 
combined group (Figure 5-1) as well. This is due, in part, to interviewee number two’s higher 
importance for safety and a lower importance for crew productivity than other members of the group: 

Level II Interviewee Candidate 

Trade-off Weights/Importance Rank Attribute States 1 

Attribute 1 2 3 PRO KNO 


Crew Productivity 

0.65/2 

0.30/3 

0.50/1 

7.5 

5.0 

Safety 

0.55/4 

0.50/1 

0.40/2 

3.3 

5.5 

Growth Potential 

0.60/3 

0.05/4 

0.25/4 

7.0 

6.0 

Spinoff Potential 

0.25/8 

0.05/4 

0.25/4 

7.5 

1.5 

Initial Cost 

0.50/5 

0.50/l 

0.25/4 

5.0 

8.0 


The attribute states for PRO indicate a low safety contribution, but high marks for growth and spinoff 
potential-attributes of least importance to the interviewee number two. The result is a much lower 
ranked PRO than the other interviewees. The KNO case is misleading because although the weights 
between individual number one and two are the same, initial cost is ranked fifth and first, respectively. 
Thus, the high attribute state for initial cost receives substantially more weight by individual number 
two, resulting in a higher ranked KNO than that of interviewees one and three. 


1 CDF's represented by mean value. 
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The above description typifies the discussion used to explain why an alternative occupies a 
particular position in the rankings. The focus of attention is now directed at a subset of the high- 
leverage prototype A&R candidates near the top of the list. Because no information was available on 
actual budget or number of high-leverage prototype AfcR candidates to be selected, the top ten options 
were chosen for closer examination. 

The problem raised by Figure 5-1 is addressed in Figure 5-5. The rankings are displayed by 
group in two ways. A “flow” diagram is used to present the group rankings under two decision rules 
each represented by a column on the left and right. Beginning at the top, the options are ordered from 
most preferred to least preferred. The first decision rule selects preferred high-leverage prototype A&R 
candidates based on their value only (the median of the outcome expected utility rankings for the 
individuals in the group). Using this rule, the user follows the left-hand path down the list. The 
second decision rule selects preferred options based on their value first, and in the event of ties, on 
minimum variance. Using the second rule, the user follows the right-hand path down the list. Note 
that in some places, one column indicates an identical ranking under both decision rules. 

Figure 5-5 shows similar concordance between the Level I and the third party group— the Level II 
group reverses the order of DIA and STD for reasons as discussed earlier. An indicator of the 
robustness of the model is the commonality of the top ten lists (the same set of ten high- leverage 
prototype A&R candidates appears in all four lists). Furthermore, the top three high-leverage 
prototype A&R candidates are the same in all four lists, DIA, STD, and KBS. 

It is interesting to note that the top three high-leverage prototype A&R candidates are software- 
related to the fields of artificial intelligence and expert systems. Two of the applications are error/fault 
detection systems and the third is a study of standards and tools to facilitate expert system 
development. 

C. RESULTS OF VARIATIONS ON THE NOMINAL CASE 

A series of runs with the multiattribute nominal, best-, and worst-case data were made to 
examine variations in the nominal results to (1) deterministic rankings; (2) the preferences of the 
interviewees (best and worst cases); and (3) the additive decision model form. The nominal case 
described above is used as the basic model (using the probabilistic case with nominal interviewee 
preference data and multiplicative decision model). The first comparison involves the nominal case 
versus all the deterministic cases. The second comparison involves the nominal case versus all the best 
and worst case options. The third comparison involves the nominal case versus all the additive 
decision model cases. Table 5-1 displays the variety of combinations for analysis. 

All comparisons are made using the rank sum rule outcomes to reduce computational complexity 
and highlight any large effects. The rank sum rule is applied across all nine individuals in the three 
groups to summarize major effects (the discussion of group differences is presented in section D). The 
basis for computing the rank sum rankings is integer ranking value. 

1. Probabilistic Versus Deterministic Results 

Table 5-2 displays the nominal rankings with differences in numerical rank from the six 
deterministic cases. Thus a blank space indicates no difference between the nominal case and the 
corresponding comparison case. A positive number indicates the specific high-leverage prototype A&R 
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Table 5-1. Sensitivity Analysis Combinations (Common Dimensions in Boxes) 


Interviewee Responses Uncertainty Model Decision Model 


(Nominal, Best, Worst) (Deterministic or Probabilistic) (Multiplicative/Additive) 

Nominal Case Ranking 





Nominal 

Probabilistic 

Multiplicative 

Deterministic Cases 





Nominal 

Deterministic 

Multiplicative 


Best 


Deterministic 

Multiplicative 


Worst 


Deterministic 

Multiplicative 


Nominal 

Deterministic 

Additive 



Best 


Deterministic 

Additive 



Worst 


Deterministic 

Additive 


Nominal vs. Best and Worst Cases 





Best 


Probabilistic 

Multiplicative 


Best 


Probabilistic 

Additive 



Best 


Deterministic 

Multiplicative 


Best 


Deterministic 

Additive 



Worst 


Probabilistic 

Multiplicative 


Worst 


Probabilistic 

Additive 



Worst 


Deterministic 

Multiplicative 


Worst 


Deterministic 

Additive 


Nominal vs. Additive Decision Model 





Nominal 

Probabilistic 

Additive 



Best 


Probabilistic 

Additive 



Worst 


Probabilistic 

Additive 



Nominal 

Deterministic 

Additive 



Best 


Deterministic 

Additive 



Worst 


Deterministic 

Additive 









2 DET denotes deterministic case; NOM, BEST, and WORST denotes nominal, best, and worst case 
interviewee data; MULT denotes multiplicative model; and ADD denotes additive model. Blanks 
represent agreement with Nominal Case ranking. 
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candidate moved up in the rankings by that value, and a negative number indicates the candidate 
moved down in the rankings from the corresponding nominal value. The number indicates how far 
the high-leverage prototype A&R candidate moved up or down. Examining Table 5-2 reveals how 
little the deterministic decision model assumption affects these results— at most, two positions for 9 of 
the 126 possible positions and one position for 26 of the 126 positions (91 positions remained 
unchanged by the decision model assumption variations). 

The majority of ranking changes (10) occurs in the deterministic worst case/multiplicative model. 
The second model with the largest number of changes (7) is the deterministic worst case/additive 
model, suggesting that the worst case interviewee responses may be the root causes of the differences. 
These results indicate a number of the interviewees believed that their preference for the state of an 
attribute was affected more if the other attributes were at a worst state than at their best state. 

The concordance shown at the bottom of the table measures the degree of agreement between 
rankings. The upper concordance among individuals is the degree of agreement across all nine 
individual rankings. A “Yes” indicates the null hypothesis of independence of rankings could not be 
rejected at a 95% level of significance. Note that all such tests performed in the study use a 95% 

significance level. The lower concordance value is the degree of agreement between the nominal case 

and each decision model. As shown, there is strong agreement for the rankings as a whole, although 
the DET/WORST/ADD case is the lowest. (Also note that the concordance between rankings for the 
Nominal Case will always be 1.0 by definition.) 

2. Nominal Versus Best and Worst Cases 

Table 5-3 displays the results of the comparisons between the nominal case and the best 
case interviewee responses. Again, there is, at most, an exchange of two positions (2 out of 84 possible 
occurrences), an exchange of one position in some cases (16 out of 84 possible occurrences), and no 
change in the majority of cases (66 out of 84 possible occurrences). Overall, the ranking positions of 
18 out of the 84 possible positions are affected. 

Table 5-4 displays the results for the comparison between the nominal case and the worst case 

interviewee responses. The differences between best and worst case responses identified in the 

deterministic case earlier are evident here. While the number of changes in ranking position is at most 
two (for 12 out of 84 possible occurrences) and in some cases one (for 24 out of 84 possible 
occurrences), there is clearly more shifting due to the worst case data (36 out of 84 possible positions). 
This can also be seen in all concordance values when compared to the best case (see Table 5-3). 

3. Comparison of Decision Models 

Table 5-5 displays the results for the comparison between the nominal case and the 
additive decision model. There is, at most, an exchange of two positions (7 out of 126 possible 
occurrences), an exchange of one position in some cases (37 out of 126 possible occurrences), and no 
change in the majority of cases (82 out of 126 possible occurrences). Overall, the ranking positions of 
44 out of the 126 possible positions are affected. 
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Table 5-3. Comparison of Nominal to Best Case Results 3 


Difference in Ranking from Nominal Case 



i 

BEST 

BEST 

BEST 

BEST 


Nominal PROB 

PROB 

DET 

DET 

Candidates 

Case 

MULT 

ADD 

MULT 

ADD 

DIA 

1 





STD 

2 





KBS 

3 





ADA 

4 

-1 


-1 


EXT 

7 


+ 1 



AUT 

9 




+i 

ROB 

4 


-1 


-1 

FAU 

10 





MNT 

8 





PRO 

6 




+i 

EVA 

11 





ES/ 

13 





SIM 

16 


+ 1 

+ 1 

+i 

POS 

12 





ABD 

15 

+ 1 

+ 1 

-1 

+i 

KNO 

14 

-1 

-2 


-2 

ACC 

18 





ARP 

19 





WOR 

21 





VDD 

20 


-1 



STE 

17 





Concordance Among 





Individuals: 

0.93 

0.95 

0.96 

0.94 

0.95 

Agreement: 

Yes 

Yes 

Yes 

Yes 

Yes 

Concordance Between 





Rankings: 

1.00 

0.999 

0.998 

0.999 

0.998 

Agreement: 

Yes 

Yes 

Yes 

Yes 

Yes 


3 DET denotes deterministic case; NOM, BEST, and WORST denotes nominal, best, and worst case 
interviewee data; MULT denotes multiplicative model; and ADD denotes "additive" model. Blanks 
represent agreement with Nominal Case ranking. 
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Table 5-4. Comparison of Nominal to Worst Case Results 4 


Difference in Ranking from Nominal Case 


Candidates 

WORST 
Nominal PROB 
Case MULT 

WORST 

PROB 

ADD 

WORST 

DET 

MULT 

WORST 

DET 

ADD 

DIA 

1 





STD 

2 


-1 


-1 

KBS 

3 


+ 1 


+ 1 

ADA 

4 

-2 


-2 


EXT 

7 

-1 




AUT 

9 





ROB 

4 

-1 

-2 

-1 

-2 

FAU 

10 





MNT 

8 

+ 1 


+ 1 


PRO 

6 

+2 

+ 1 

+2 

+i 

EVA 

11 





ES/ 

13 

+ 1 

+ 1 

+ 1 

+i 

SIM 

16 


+ 1 



POS 

12 

-1 

-1 

-2 

-2 

ABD 

15 


+ 1 


+2 

KNO 

14 


-2 

+2 


ACC 

18 

-1 


-1 


ARP 

19 

+ 1 


+2 


WOR 

21 





VDD 

20 


-1 



STE 

17 



-1 


Concordance 

Among 





Individuals: 

0.93 

0.94 

0.95 

0.94 

0.95 

Agreement: 

Yes 

Yes 

Yes 

Yes 

Yes 

Concordance Between 
Rankings: 1.00 

Agreement: Yes 

0.996 

Yes 

0.996 

Yes 

0.996 

Yes 

0.991 

Yes 


4 DET denotes ^ deterministic case; NOM, BEST, and WORST denotes nominal, best, and worst case 
interviewee data; MULT denotes multiplicative model; and ADD denotes “additive'' model. Blanks 
represent agreement with Nominal Case ranking. - 











5 DET denotes deterministic case; NOM, BEST, and WORST denotes nominal, best, and worst case 
interviewee data; MULT denotes multiplicative model; and ADD denotes additive model. Blanks 
represent agreement with Nominal Case ranking. 
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4. Work Package Overlaps 

Because of the proposal submission process, areas of overlap exist, not only across the high- 
leverage prototype A&R candidates, but also between some of the candidates and the S.S. Freedom 
Program Work Packages (References 48 through 51). Before a high-leverage prototype A&R candidate 
can be recommended for funding, it is reviewed for both types of overlap. When areas of overlap are 
identified, reductions in scope may or may not be required. Overlap in some areas may, in fact, be 
necessary to reduce risk or to provide management with alternative approaches. In such cases, 
reductions in scope would not be required. Other overlap areas may be eliminated to avoid duplication 
or control cost. Reductions in scope are recommended in such cases. Such decisions must be made on 
a case-by-case basis. One of the purposes of the “DECISIONS TO FUND” box is to make such 
determinations in light of technical risks and budget constraints. 

Three courses of action are possible. Surviving proposals can: (1) be incorporated into rescoped 
Work Packages; (2) be incorporated into existing (unmodified) Work Packages; or (3) be funded 
under separate contract. This process is illustrated in Figure 5-6. All proposals were reviewed and 
Table 5-6 illustrates both types of overlap and recommendations. 



Figure 5-6. Process for Resolution of High- Leverage Prototype Overlaps 
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Table 5-6. Areas of Work Pack age/ High- Leverage Prototype Overlap 


IDENTIFIER TITLE 

NOTES 

DIA Diagnostic EPS 

Proposal for a hierarchical diagnostic ES 6 for EPS integrated with 
DMS* Some overlap with WP4 (see Reference 48 pp. A- 13 to A-26). 
Should be integrated with WP4. 


STD Stds/Tools for ES 

No apparent overlap. 

KBS Fault redundancy 

WP2 proposed ES for FDIR in DMS/OMS plans 8 (see Ref. 49 pp. 1- 

management 

99 to 1-104). 

ADA ADA Effectiveness 

WPs 2 (see Ref. 49, p. 1-99), 4 accept ADA as the given language and 
are proceeding with software development. Therefore the issue of ADA 


effectiveness is moot. 

EXT Extended fault- 


tolerant testbed 

No apparent overlap. 

AUT Auto Robotic Assy 

Some overlap with ROB. Both propose a facility that emulates shuttle- 
based control of robot(s) for assembly; should be integrated with each 


other and ultimately with WP 3 (see Reference 51). 

ROB Robotic control 
SS Assembly 

Some overlap with AUT. Both propose a facility that emulates shuttle- 
based control of robot(s) for assembly; should be integrated with each 

other and ultimately with WP 3 (see Reference 51). 

FAU Fault Prediction 

Identifies fault prediction of electrical equipment as goal of KBS. WP 4 

for Electrical 

identifies ES in baseline configuration (see Ref. 48, pp. A-l to A-25). 

Equipment 

Candidate would transfer on-line predictive system to S.S. Freedom 
Program. Should be integrated with DIA and then WP 4. 

MNT KBS 9 Develop- 

Some overlap with KNO, ES/. Any proposal to define KBS standards 

ment and 

must necessarily include knowledge acquisition (KA) standards. KBS 

Maintenance 

development maintenance may also overlap the ES standards research. 


6 ES: Expert System. 

7 EPS: Electric Power Subsystem; DMS: Data Management Subsystem; WP 4: Work Package 04 
(Reference 48). 

8 FDIR: Fault Detection, Isolation, and Recovery; OMS; Operations Management Subsystem; WP 2: 
Work Package 02 (Reference 49). 

^Knowledge-Based System. 
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Table 5-6. Areas of Work Package/High- Leverage Prototype Overlap (Continued) 


IDENTIFIER TITLE NOTES 

PRO Prototype 
Scheduler 

Some overlap with WP 2 (see Reference 49, pp. 1-102 to 1-103). 

EVA EVA Retriever 

Dropped from WP contracts; NASA in-house effort. 

ES/ ES Stds Research 

Some overlap with MNT, KNO. Any proposal to define KBS standards 
must necessarily include KA standards. KBS development maintenance 
may also overlap the ES standards research. 

SIM Auto Robotic 
Simulation 

No apparent overlap. 

POS Telerobotic and 
Thermal Posture 
Predictor 

WP 2 includes thermal prediction model (see Reference 49, p. 1-32). 
Use of FTS 10 is unclear. 

ABD Abductive Tech. 

No apparent overlap. 

KNO Knowledge 
Acquisition 
Standards 

Some overlap with ES/, MNT. Any proposal to define KBS standards 
must necessarily include KA standards. KBS development maintenance 
may also overlap the ES standards research. 

ACC Accel SSM Demo 

Acceleration of existing effort; no apparent overlap. 

ARP A&R Prototyping 

WP 1 lists as part of A&R: risk-level BIT/BITE on-board maintenance 
system (see Reference 50, p. 1-59). 11 

WOR World Modeling 

No apparent overlap. 

VDD VDDTtoOMA 

No apparent overlap. 

STE Stereo 

Telepresence 

No apparent overlap. 


10 Flight Telerobotic Servicer. 

11 WP01: Work Package 1 (Reference 50); BIT/BITE: Built-in Test/Built-in Test Equipment. 
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The following brief summary is helpful in relating Work Package responsibilities to high-leverage 
prototype AfcR candidate overlap areas. 

A number of potential overlap areas have been identified. In most cases the information in the 
proposals is inadequate to provide full resolution. Before a final assessment of overlap can be 
completed, fact-finding sessions are required. Proposers would be required to resubmit additional 
information in the indicated areas. Table 5-7 summarizes the situation at the present time. 


Table 5-7. Work Package Overlaps 


Candidates Degree of Overlap Identified 



Possible/ 

None 



Definite 

Potential 

Apparent 

Unknown 

Moot 

DIA 

KBS 

EXT 

STD 

ADA 

AUT 

ROB 

FAU 

MNT 

PRO 




EVA 

ES/ 


SIM 



POS 


ABD 



KNO 

ARP 

ACC 





WOR 

VDD 

STE 




There are a number of potential overlaps among the submissions. There are also large 
inconsistencies in quality and completeness among the alternative high-leverage prototype A&R 
candidates, particularly in coordination with the Work Packages. These potential overlaps are 
identified and recommendations made for resolution. In some cases, a high-leverage prototype A&R 
candidate is recommended for incorporation into a Work Package. For these cases, the choice between 
direct incorporation and increased contract scope is a decision for the cognizant Work Package Center. 

5. Comparison to Working Group Evaluations 

During the course of the study, the AAWG, RWG, and AIESTWG conducted independent 
evaluations of the high-leverage prototype AfcR candidates within their respective areas. The working 
group evaluations used a variety of criteria and weighting procedures different from the methodology of 
this study. Thus, it is easier to assess where differences exist rather than why they exist. However, 
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some general observations can be made. Figure 5-7 summarizes the working group results and the 
study results. The candidates enclosed in boxes represent matches with the top ten high-leverage 
prototype A&R candidates obtained by the study methodology. While there are agreements between 
the top ten list and the working group list, there are also discrepencies. It is impossible to ascertain 
what the working group rankings might have been for a merged list, however, it is reassuring that 
numerous matches do exist. 

D. CONCORDANCE OF RANKINGS 

This section presents and discusses the results of the concordance calculations for the rankings 
presented earlier (see Section III-C for a discussion of concordance statistics). Three different types of 
concordance were calculated and analyzed: 

(1) Individual rankings within groups. 

(2) Group rankings with different decision rules. 

(3) Group rankings using different multiattribute decision model assumptions. 

The purpose of the concordance analysis is to assess the sensitivity of the rankings to differences among 
individuals within groups; differences in group decision rules; and variations in decision model 
assumptions regarding the decision models used. The results are summarized in Table 5-8. Overall, 
the rankings are highly concordant for all three cases implying that the rankings presented earlier are 
robust under a variety of conditions. 

The concordance of the rankings given to the twenty-one alternative high-leverage prototype A&R 
candidates by individuals in the four groups studied was examined in two ways: (1) by comparing the 
individual rankings within groups for each of the 48 computer runs described earlier in this section; and 
(2) by comparing the group rankings according to the additive, Nash, and rank sum rules (see Table 
5-8). The following observations can be made: 

(1) There is not perfect agreement throughout the rankings. 

(2) There are several instances where the ranks assigned to several alternatives by one 
interviewee in a group appear to be at variance from those given by the other interviewees 
in the group, particularly for the Level II group. Because the lowest concordances in the 
table are between the deterministic and probabilistic nominal cases, this may be 
attributable to differences in attitude toward risk. 

(3) The concordance measures are in every instance, highly significant. Each case is 
statistically significant below the 5% level. 

(4) A detailed comparison of rankings was performed to examine the effects of varying the 
decision model assumptions (see Section C). While the model was still robust and 
agreement obtained, the WORST case option has the lowest concordance within the 
variety of decision model assumptions studied. As in (2) above* this may be due to the 
interviewees’ risk aversion to an indifference point when all the remaining attributes are at 
their worst state. However, the fact remains that although the risk aversion may have 
been an effect, it is not significant at the 95% level. 


r 


i 
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HIGH-LEVERAGE PROTOTYPE EVALUATION 

COMBINED GROUP COMPARISONS: DETERMINISTIC VS. 
PROBABILISTIC CASES AND VS. WORKING GROUPS 



Figure 5-7. Comparison to Working Group Rankings 
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Table 5-8. Concordance Analysis Results 


HIGH-LEVERAGE PROTOTYPE EVALUATION 
CONCORDANCE ANALYSIS 

D/P: DET/PROB 

N/B/W: NOM/BEST/WORST CASE 
M/A: MULT /ADD DECISION MODEL 

OVER 4 GROUPS - 2*3*2*4-48 CASES 



GROUP 1 

GROUP II 

GROUP III 

ALL 


1 

G 

1 

G 

1 

G 

1 

G 

D/N/M 

0.97/A 

1 .00/A 

0.90/A 

1 ,00/A 

0.96/A 

1 .OO/A 

0.93/A 

1 ,00/A 

P/N/M 

0.98/ A 

1 .00/A 

0.90/A 

0.99/A 

0.97/A 

1 .OO/A 

0.93/A 

1 .OO/A 

P/N/A 

0.98/A 

1 .00/A 

0.92/A 

1 ,00/A 

0.97/A 

1 .OO/A 

0.95/ A 

1 .OO/A 

D/N/A 

0.92/A 

1 .00/ A 

0.97/A 

1 .00/A 

0.96/A 

1 .OO/A 

0.94/A 

1 ,00/A 

D/B/M 

0.92/A 

0.99/A 

0.98/A 

1 .00/A 

0.97/A 

1 .OO/A 

0.94/A 

1 ,00/A 

D/W/M 

0.95/A 

0.99/A 

0.95/A 

1 .OO/A 

0.97/A 

0.99/A 

0.94/A 

1 .OO/A 

P/B/M 

0.93/A 

0.99/A 

0.98/A 

1 .OO/A 

0.97/A 

1 OO/A 

0.95,/A 

1 .OO/A 

P/W/M 

0.95/A 

0.99/A 

0.96/ A 

1 .OO/A 

0.97/A 

1 .OO/A 

0.94/ A 

1 .OO/A 

D/B/A 

0.94/A 

1 .00/A 

0.90/A 

1 .OO/A 

0.97/A 

1 .OO/A 

0.95/A 

1 ,00/A 

D/W/A 

0.95/A 

1 ,00/A 

0.90/A 

1 .OO/A 

0.97/A 

1 .OO/A 

0.95/A 

1 .OO/A 

P/B/A 

0.9 5/A 

1 .OO/A 

0.92/ A 

1 .OO/A 

0.98/A 

1 .OO/A 

0.96/A 

1 .OO/A 

P/W/A 

0.96/A 

1 ,00/A 

0.97/A 

1 .OO/A 

0.98/A 

1 .OO/A 

0.95/A 

1 .OO/A 
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(5) As a result, by each of the comparison methods, there is substantial agreement in the 

rankings of the twenty-one high-leverage prototype A&R candidates within each of the four 
groups analyzed and for the variations in decision model assumptions performed. 

It should be noted that the high levels of concordance should not be inferred to mean there are no 
disagreements in rankings. As shown earlier in Tables 5-2 through 5-5, small numbers of position 
changes do occur. The task remains for the Associate Director to weigh the effects of these small 
changes together with the degree of work package overlap and any other programmatic considerations 
before selecting a recommended set of high-leverage prototype AfcR candidates. 
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SECTION VI 

DISCUSSION AND CONCLUSIONS 


A. DISCUSSION 

To place the results of the study in context with the Space Station Program, two issues should be 
considered: 

(1) What are the goals (values to be maximized) that should be used to evaluate the high- 
leverage prototype A&:R candidates? 

(2) What steps need to be taken to implement the study results within the S.S. Freedom 
baseline? 


The purpose of high-leverage prototype AfcR candidates is to provide near-term, low-risk 
capabilities for inclusion in the baseline S.S. Freedom. With respect to the first issue, if the value to 
be maximized is high-value with high-risk, then a different perspective and rank ordering is implied 
(see Figure 6-1). A legitimate role for NASA could be the subsidization of risky technology 
developments with high payoffs. Such an approach could yield larger potential benefits for commercial 
spinoffs than the alternatives implied by this study. If the low-risk definition of a high-leverage 
prototype is maintained, the approach followed here is appropriate. However, if the restriction that the 
technologies be baselined is relaxed and the evolution-phase is considered, long-term “risky technology 
developments could be subsidized and candidates with larger variances could then be included. 


HIGH -LEVERAGE PROTOTYPE EVALUATION 
WHICH PROPOSALS SHOULD BE SELECTED? 

XX 

HIGH-VALUE/LOW RISK HIGH-VALUE/HIGH RISK 


PRO CON 


PRO CON 


CONTRACTORS LIKE 
THESE (LOW RISK) 

GOOD INVESTMENT 


HIGH PROBABILITY 
OF SUCCESS 


LIMITS CREATIVITY 


MAY BE INVESTING IN 
WRONG TECHNOLOGY PATH 
FOR LONG-TERM OPS. 

TECHNOLOGY MAY BE 
ACHIEVABLE BUT NOT 
COMPETmVE (POWER. 
MASS) 


PROVIDES SUPPORT TO 
RISKY /VISIONARY 
PROJECTS 

POTENTIAL FOR HIGH 
RETURN ON INVESTMENT 

TRAINiNC/EDUCATION 

VALUE 

SKILLBASE 

ENHANCEMENT 


HIGH OVER HEAD- -RISKY 
/VISIONARY PROJECTS 

LOWER PROBABILITY OF 
SUCCESS 

LESS SUPPORT FOR 
mainstream needs 

POTENTIAL LACK 
OF SKILLBASE 


Figure 6-1. Alternative High- Leverage Prototyping Program Goals 
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The second issue is one of logistics. After a final set of high-leverage prototype A&R candidates 
is established, candidates with work package overlaps may need rescoping. In addition, if any high- 
leverage prototype A&R candidates are being funded from other sources, at the decision point to select 
a high-leverage prototype, the funding for that source should be coordinated between the appropriate 
S.S. Freedom offices to formalize management control and responsibility of the technology 
development. '' 

As a process, the approach of this study required a number of adaptive “adjustments.” The aim 
of the process was to facilitate the programmatic evaluation of A&R candidates to relieve some of the 
overload at the working group level and ensure consistent, objective evaluation. However, due to 
internal working group difficulties, the attribute state data required for this study were developed by a 
third party technical committee. There were additional complications in task progress due to frequent 
management turnover within the S.S. Freedom Program (3 Associate Administrators and 5 Directors 
during the course of the study). 

Irrespective of the turbulence in the external environment, the process did move forward. Was 
the process a good one? Was the process effective in meeting the objectives outlined in Section I? To 
address some of these issues, a post-process assessment questionnaire was mailed to the participants 
after the results had been presented (see Appendix D). While approximately one-third of the twenty 
participants responded (30%), it should be noted that the number of returned questionnaires may be 
insufficient to draw definitive conclusions. Table 6-1 summarizes the responses received. Overall, the 
results indicate a majority of responses with a good-to-excellent rating, however, there were a number 
of problem areas needing improvement, 

On the positive side, there was general agreement that the process of ranking was useful both for 
improving understanding and evaluation. Four participants responded that the process would be useful 
in future evaluations, while two felt it would not. Intuitive concurrence with the results was rated 
good with one exception. Of the four who answered the question, antigaming safeguards were judged 
fair-to-excellent. 

On the negative side, it was mentioned that the proposers of the high-leverage prototype A&R 
candidates were not represented well by their submitted proposals alone. This belief almost certainly 
resulted from the informal nature of the call for high-leverage prototype A&R candidates and delays in 
their evaluation. If the study had been intended solely as an exercise, then the usual protocols 
restricting contact between evaluator and proposer would most likely have been eased or eliminated. 

The reasonableness of the ground rules was judged as good, and the handling of uncertainties was 
also above average. The overall credibility of the technical assessment process, however, was judged as 
only average-perhaps due to concerns about the informality of the original call. Finally, the group 
decision rules were judged to be good. 

There was no general agreement on the preferred process for future evaluations, but the results 
(five out of six) indicate a desire to have some type of computerized decision support capability for the 
participants. Comments about the use of argumentation analysis regarded the process as useful. 
However, the analytic, labor-intensive nature of describing each argument component was a tedious 
process that could be facilitated by the use of a software database tool to enter, edit, and manage the 
large quantities of text information. A number of comments reflected a sense that the argumentation 
analysis provided a balance between the quantification of the vast array of numerical attribute state 
inputs and thinking behind each high-leverage prototype A&R candidate. 
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Table 6-1. Post-Process Questionnaire Results 


A. YOUR OVERALL REACTION TO THE EVALUATION Unsatisfactory Poor Fair Good Excellent NA 1 

(1) Describe the value of ranking the A&R candidates 

in Improving your understanding of them. □ □ 2 3 1 □ 

(2) Describe the value of rankings for evaluating A&R candidates. □ □ 1 2 3 □ 

(3) Describe the usefulness of the results for measuring candidate 

value. D 1 2 3 D D 

(4) Describe the effectiveness of the safeguards in preventing 

"gaming” by the participants. □□2112 

(5) Describe the fairness of the process from the proposers' 

point of view. 0 2 2 2 0 0 

(6) Describe the fairness of the process from your 

point of view. 0 1 1 4 0 0 

(7) Describe the efficiency of the process for generating final 

rankings □ 1 12 2 0 

(8) After examining the results, describe your intuitive 

concurrence. 1 0 1 3 1 0 

(9) Describe the utility to you of similar processes in future 

evaluations. 1 1 0 2 2 □ 


B. THE TECHNICAL ASSESSMENT PROCESS 

(1) Describe the reasonableness of the ground rules. 

(2) Describe the adequacy of the process in addressing the 
uncertainties in candidate attributes. 

(3) Describe the validity of using arguments for and against 
the attribute ratings in establishing technical credibility. 

(4) Describe the credibility of the technical assessment 
process. 

(5) Describe the adequacy of the technical assessment 
process in preventing "gaming.” 


□ 3 


□ 2 


0 3 


C. THE ASSESSMENT OF PRIORITIES 

(1) Describe the adequacy of SSF Decision Maker 
representation, given the scope of the evaluation. 

(2) Describe the utility of the group decision rules for 

expressing group preferences. 


TOTAL RESPONSES: 
PERCENTAGE: 


2 10 25 36 16 7 

2% 10% 26% 38% 17% 7% 


(3) If asked to participate in a future evaluation, would you prefer a process with: 

Analyst support only? Interactive computer support Stand-alone computer software 
with analyst backup? with analyst coordination? 

1 response 3 responses 2 responses 


l NA: Not applicable or no response. 
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Objections to the methodology were not specific to the multiattribute decision analysis model or 
framework other than the comment mentioned in Section IV, but were aimed at the potential of 
participants to bias the results. These concerns are common to any type of evaluation using virtually 
any model. However, the original objectives were to develop: 

(1) 

( 2 ) 

( 3 ) 

( 4 ) 

It is apparent that a consistent basis for evaluation was developed and applied to a set of A&R 
candidates. The approach is consistent in the sense that the same decision model was applied to each 
high-leverage prototype A&R candidate in the same computational manner. 

A mechanism was established to confirm the analysis of high-leverage prototype A&R candidate 
advocates. There were some disagreements between the working group evaluations and the study 
results, but the results match for the majority of high-leverage prototype AfcR candidates. 

A collection and aggregation of input was obtained from concerned parties based on a common 
set of program evaluation attributes. Whether all the concerned parties were represented is a matter 
for debate. Additional time might have allowed a separate interested party group for the operations 
perspective, however the scope and timing of the analysis and resources involved did not indicate a 
need for a large number of interest groups during the course of the study. There were a number of 
comments made prior to the application of the process: 

Comment (1): 

" The evaluation process should allow those who assign ratings for the attributes to choose a range of 

values within which the actual value is felt most likely to reside. The proposals and the criteria are 

both difficult to quantify and the sensitivities of the evaluation results to variations in attribute scores 

should be made visible in the final output (Reference 52)/’ 


A consistent basis for evaluation. 

A mechanism for reaffirming or checking the analysis of the A&R candidate 
advocates. 

An aggregation of input from all concerned parties, based on a consistent 
set of program evaluation attributes. 

A trial application of a proven tool for the A&R implementation and 
evaluation process. 


Comment (1) refers to the issue of transparency— that the process allow the users to provide 
estimates in an understandable format that conveys the meaning and the message. If a numerical 
value cannot be estimated, it should be acceptable to indicate such a response and continue with the 
process. Furthermore, the sensitivities of the results to variations in attribute ratings and interviewee 
responses should be displayed in an intuitive manner. The sensitivity analysis approach examines 
variations in the decision model and input data sets using statistical concordance of rankings. As 
shown in Section V, there is sensitivity to changes in model assumptions but not statistically 
significant sensitivity. This does not imply that such changes (at most two ranking positions) could 
not be important. Depending on how resources are allocated, such differences could be significant to 
high-leverage prototype A&R candidates at the margin— those candidates that might not be selected 
due to a change in ranking of two positions. While budgeting strategies and RfcD portfolio 
considerations are beyond the scope of this study, it is possible to devise strategies for dealing with such 
issues (e.g., building redundancy into the program to reduce risk by splitting and redistributing 
resources across the marginal projects). 
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Comment (2): 

" The true value of your analysis methodology is in the iteration with the proposal evaluators to force 
them to think about the real value of the proposal as opposed to the "technical beauty" of the 
proposal. I am sure we all agree that the output of a computer algorithm should not be used to select 
the proposals for funding without further "sanity” checks (see Reference 52). H 

Comment (2) is a reminder that people, not computers and algorithms, make the important 
decisions. This fact was recognized at the outset of the study and reaffirms the need for the 
decision maker to synthesize not only the information contained in this study and degree of work 
package overlaps, but also the programmatic risks and uncertainties and other factors impinging 
on the selection of an appropriate set of high-leverage prototype A&R candidates. 

Comment (3): 

"... The major difficulty is that there will be either insufficient knowledge or proposal content to enable 
the reviewers to accurately position each proposal in the set of attributes that you have provided. I 
think the exercise will enable us to see just how much subjectivity is present in an evaluation of this 
type. Also, I think it is valuable to see the variation in answer ranges for each of the people rating the 
proposal to get some idea of the degree of consensus (or lack thereof). Means/averages are not 
sufficient. Through several iterations with the same group of reviewers, I think we can refine things to 
the point that we might become comfortable with the process (Reference 53)." 

Comment (3) again raises the issue of high levels of uncertainty inherent in the set of high- 
leverage prototype A&R candidates studied and how variations in their attribute states impact the 
resulting rankings. As described in this publication, the methodology presented provides a way of 
incorporating multiple attributes with different levels of uncertainty to allow the use of actual 
cumulative probability distribution functions and thus characterize the impacts of a broad array of 
uncertainties on the final ranking outcome. These variations are shown to have minimal impact on 
the final ranking values and consensus for the case study presented herein. However, for other 
applications, these impacts would be quantifiable and traceable for cases where consensus did not exist. 

The main theme of contention with the approach is a “concern” that the inputs of each 
participant are reviewed and available to the other participants as a method of checks and balances to 
ensure that biases are minimized and fairness prevails. Whether this reflects a commonplace desire to 
simply understand the process in a transparent manner is unclear. However, the issue is relevant for 
future studies and evaluations and is discussed in Section 6-C. 

B. PROGRAMMATIC IMPLICATIONS OF THE RESULTS 

The analysis has four implications based on interviews with NASA personnel, the analysis 
methods, and the results obtained. 

The first implication is that high-leverage prototyping is beneficial to the Space Station Freedom 
Program as a means for transferring technology from the advanced development program to the 
baseline program. The gap between the research and implementation phases of a technology has 
traditionally presented both a barrier and an opportunity to the utilization of that technology. The 
barrier results from the lack of a clear and credible message from the technologist to the potential user 
that the technology already has clear applications, rather than being a solution in search of a problem. 
While most technologists would claim this for their own technology, there is such a barrage of these 
messages that the user must have some reliable means to sift through them. The opportunity is that a 
program manager can allocate resources to a high-leverage prototyping program, selected according to 
the method outlined in this report, that will reveal and foster those technologies that have higher 
maturities and probabilities of success. 
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The second implication is the study provides guidance as to which areas for the set of twenty-one 
high-leverage prototype AfcR candidates are promising. They include: 

DIA Diagnostic Expert System for electric power and data management 

STD Standards and Tools for Expert Systems 

KBS Knowledge-based system for fault detection in data management and 

operations management systems 
EXT Testbed for extended fault- tolerant testing 

AUT Study of robotics for assembly and fasteners with testbed 

FAU Knowledge- based system for electric power subsystem fault prediction 

MNT Study of knowledge-based system development and maintenance tools 

ROB Software upgrades to demonstration of robotic control of assembly 

PRO Prototype crew scheduler using advanced search methods 

Thus, even if a high-leverage prototyping program does not become an on-going process, the above 
technologies could make a positive contribution to the baseline S.S. Freedom . There will always be 
some hesitation to accept automatically, the results of such evaluations. Uncertainties necessarily play 
a major role in any selection. Reviewers must pay particular attention to this issue. Examining the 
above list, two commonalities are apparent. 

The first commonality is that virtually all of these recommended high-leverage prototyping A&R 
candidates involve software (DIA, STD, KBS, FAU, MNT, ROB, and PRO) as opposed to hardware 
elements (EXT, AUT). The software emphasis focuses primarily on the application of Expert System 
and Knowledge-Based Systems to S.S. Freedom subsystems with high potential for early benefits: the 
electric power, data management, and operations management subsystems. This may be due, in part, 
to earlier studies indicating the application potential of these technologies and subsystems (Reference 
54) cited by a number of the high-leverage prototyping A&R candidates. A secondary, but important, 
emphasis in the list involves the study of support tools for the development of Expert and Knowledge- 
Based Systems (STD, MNT). 

The second commonality is that virtually all of the high-leverage prototyping A&R candidates 
involving applications to hardware focus on the same subsystem areas. This was also true to some 
extent, for the full list of candidates. The most common concept was the study of some aspect of fault 
detection, fault tolerance, and fault prediction related to the electric power subsystem. There was also 
interest in applications for the data management and operations management subsystems. 

The resulting focus of the high-leverage prototyping A&R candidates on automation applications 
may be a facet of the trade-off between implementation needs and schedule risk. As stated earlier, the 
purpose of the high-leverage prototyping activity is to leverage advancing A&R technologies into the 
baseline S.S. Freedom . An implication of this study is that software (automation) candidates represent 
a class of technologies well suited for late addition to the baseline S.S. Freedom design. As a high- 
leverage AfcR candidate increasingly involves the development or testing of hardware, the time 
requirements for development can increase and pressure the decision (by increasing the schedule risk) to 
include or not include the candidate in the baseline. Such risks can be mitigated if the hooks and 
scars for the technology are in the baseline, because the technology can be added at any time. 
However, if the hooks and scars are not present, a high-leverage prototype that is not ready on time 
will need to be redesigned or (possibly) dropped from consideration. Such reasoning might explain 
why most of the recommended candidates involve software developments. It is more likely that two 
world views exist. The first world view believes that software development is inherently risky and that 
the purpose of the high-leverage prototyping program is to encourage risky prospects. The second 
world view believes that software development is low in risk and thus improves the likelihood that the 
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high-leverage prototyping A&R candidate will be implemented in the baseline S.S. Freedom design. 
Whether the different types of high-leverage prototyping A&R candidates were presorted by the 
proposers based on these different world views is unknown, because the original definition of the high- 
leverage prototyping program included both notions of investing in high-risk candidates with the 
objective of implementation in the baseline design. 

The third implication of the study is the risks and uncertainties associated with advanced 
technologies can be quantified in a manner that surfaces the relevant issues for decision making in 
those highly uncertain environments. The analysis of the decision model inputs has shown that the 
intuitively high level of uncertainty associated with the twenty-one high-leverage prototype A&R 
candidates is reflected numerically in the variance of both the attribute states and the resulting 
expected utility value used to rank order the alternatives. The question still remains whether high 
uncertainty, as exhibited by high variance, is a positive or negative feature of the high-leverage 
prototype A&R candidate. Any high-leverage prototyping program needs to understand and determine 
whether to subsidize high-value low-risk developments or high-value high-risk developments. Because 
the programmatic interest for this study is to make the high-leverage prototype A&R candidate a 
feature of the near-term baseline, pragmatic considerations favor the low uncertainty, low-risk 
candidates. However, in an environment where hooks and scars exist for high-leverage prototype AfcR 
candidates, high-risk developments would not incur the severe cost and schedule penalties of a “no-go” 
decision and could be implemented later in the evolutionary phase. A fundamental question for 
technical managers of such R&D programs is how to balance risks. For example, during the early 
stages of project development, the RfcD portfolio might consist of primarily high-risk tasks with a 
small number of low-risk tasks. As the project draws closer to finalization of the design, the mixture of 
tasks would shift toward fewer high-risk tasks with a majority of low-risk tasks. 

The fourth implication is the potential to automate and transfer various elements of the process 
using software tools to provide, for example, a distributed decision support system for use by the 
participants; a tool for group decision conferencing and post-process analysis; a tool for assisting the 
technical assessors in providing the probabilistic data for the attribute states; or a tool to facilitate the 
construction of supporting arguments for the argument analysis. Such tools can eliminate much of the 
bookkeeping associated with the databases and computations and allow the participants more freedom 
to focus on the decisions to be made. It is also important to note that the multiattribute decision 
analysis process used in this study is not constrained in any fashion to the evaluation of high-leverage 
prototypes. The same approach has been applied in many different types of technology evaluations 
and could be used in other S.S. Freedom, Lunar exploration, and Mars exploration technology-related 
decision-making contexts as such programs evolve. 

C. LIMITATIONS AND AREAS FOR FUTURE IMPROVEMENTS 

1. Limitations 

Two factors that limited the study were the inconsistencies in quality and completeness of 
data for the various high-leverage prototype AfcR candidates. There was no common format to be 
followed, inadequate data were submitted in many cases, and a number of high-leverage prototype 
AfcR candidates were not coordinated with the work packages. Because of limited resources, it was 
not possible to formally validate the data provided, hence, all attributes used in the evaluation were 
subjective and thus subject to uncertainty. Although these uncertainties are large for specific attribute 
states, they were not judged to have significant impact on the consensus of the resulting rankings. In 
fact, these uncertainties highlight the differences between a low and high-risk high-leverage prototy ping 
program and the need to determine an appropriate role for NASA for funding such an R&D program. 


H 
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During the technical assessment, a secondary set of attributes was assigned to different assessors 
for the purpose of cross-checking the attribute state ratings between different individuals. However, 
there were insufficient resources to recalculate a second analysis and examine the impacts of using these 
alternative inputs on the results. It would be useful to complete the secondary analysis as an 
alternative benchmark from which to judge the consistency of the technical assessments. 

A third factor was the scope of the interested party interviews. Although the number of 
interviewees (11) may have been appropriate for the size of the evaluation, it would have been useful to 
interview representatives from upper S.S. Freedom Program management areas and additional 
representatives from the operations areas. 

2. Future Improvements 

Four improvements to the process were identified for future applications. 

The first improvement would require the establishment of common goals and perspectives prior to 
the call for concepts. An element of setting such goals would be the development of an attribute list to 
facilitate the collection of more complete attribute information according to a common format. The 
requirements for completion of these attributes would be provided to the proposer when the call for 
proposals is made. 

The second area for improvement would be the establishment of an on-going A&R function at the 
systems engineering and integration level with a single focal point of responsibility to coordinate such 
evaluations for the variety of anticipated AfcR candidates, including the high-leverage prototype AfeR 
candidates. Should specific high-leverage prototype A&R candidates be funded and pursued, additional 
data could be used to perform more detailed analyses of the selected candidates for validation purposes. 

The third area for improvement would be the development of a group process for review and 
selection of the high-leverage prototyping A&R candidates. For the current study, this step was 
performed by the authors. However, the questionnaire respondents indicated a desire to monitor the 
data and decision algorithms during the process rather than at the end of the process when it is 
difficult to modify elements of the process. The review process could take the form of a “decision* 
conference involving the decision making participants, the analysts, and video display of the decision 
variables and data. 

A fourth area for improvement to the process would be the addition of a user-friendly decision 
support system (DSS) helpful to development, evaluation, comparison, and application of the high- 
leverage prototyping A&R candidates. The DSS could help answer a wide variety of “what-if?” 
questions and should provide transparency (or visibility) into the causes of the results. The 
argumentation analysis of the attribute state ratings would also be much easier if a simple data entry 
and database management tool were available to help the technical assessors step through each 
question. Such a DSS could support a group of decision makers in a conferencing environment with a 
single projected video display. An alternative would be a DSS used by the analysts to prepare a 
distribution file for individual participants using a microcomputer-based program and the case-specific 
data diskette. 


6-8 






Discussion and Conclusions 


D. CONCLUSIONS AND RECOMMENDATIONS 

A number of conclusions and recommendations can be drawn from this study: 


1. The Process 

(1) The first implication is that high-leverage prototyping is beneficial to the Space Station 
Freedom Program as a means for transferring technology from the advanced development 
program to the baseline program. Such a mechanism serves as a formal bridge between 
technology development programs and end-users. The bridge, however, needs to be 
specified with greater detail. 

(2) The purpose of high-leverage prototype A&R candidates is to provide near-term, low-risk 
A&R applications for inclusion in the baseline S.S. Freedom . The interest in low-risk 
development for this application is to make the application part of the near-term baseline. 
High risk developments could also be considered if evolutionary applications are included. 
Any high-leverage prototyping R&D program should determine whether to subsidize high- 
value, low-risk developments or high-value, high-risk developments. The needs in both 
cases are different. 

(3) The application of multiattribute decision analysis methods was useful for meeting the 
numerous requirements and constraints specific to the S.S. Freedom Program. In 
particular, the ability of the methodology to address multiple attributes with uncertainty, 
multiple interested parties to the decision, and monitoring of consensus was seen to be 
effective and practical. 

(4) The process would be facilitated by computer-based interactive programs to enter and edit 
the evaluation problem to the extent that major steps could be automated. Such a 
Decision Support System (DSS) for use in a distributed fashion by individuals or as a 
central display for decision conferencing by a group would allow greater flexibility for 
examining alternative assumptions. Similar software could also be developed for 
conducting the argumentation analysis of the technical assessment. 

2. The Application 

(1) Promising technology areas of potential relevance to the S.S. Freedom are: 


DIA* 

STD 

KBS* 

EXT 

AUT* 

FAU* 

MNT* 

ROB* 

PRO* 


Diagnostic Expert System for electric power and data management 
Standards and Tools for Expert Systems 

Knowledge-based system for fault detection in data management and 

operations management systems 

Testbed for extended fault-tolerant testing 

Study of robotics for assembly and fasteners with testbed 

Knowledge- based system for electric power subsystem fault prediction 

Study of knowledge- based system development and maintenance tools 

Software upgrades to demonstration of robotic control of assembly 

Prototype crew scheduler using advanced search methods 


^Possible overlap with existing work packages and/or with other high-leverage prototyping A&R candidates. 
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(2) The results of this study are supported by the separate working group evaluations although 
a number of serious differences were observed. These differences may be attributable to 
the use of primarily technical criteria as opposed to the Level II S.S. Freedom Program 
criteria outlined in this study, 

(3) The majority of recommended high-leverage prototyping A&R candidates involve the 
study or development of software either for specific applications to systems or to support 
the software development process in some manner. The high values for software-oriented 
applications may be a reflection of two world views: (i) high-leverage prototyping should 
encourage high-risk investments and software developments are perceived as risky; or (ii) 
high-leverage prototyping should encourage low-risk investments to increase the potential 
for inclusion in the baseline S.S. Freedom and software developments are perceived as low- 
risk. It is not clear whether or how these world views might have affected the types of 
high-leverage prototype A&R candidates at the point of submittal. 

(4) The applications of the recommended high-leverage prototyping A&R candidates focus 
primarily on fault detection, fault tolerance, and fault prediction software using Expert 
Systems and Knowledge- Based Systems. The target subsystems for these applications are 
the electric power subsystem, the data management subsystem, and the operations 
management subsystem. 

3. The Model Results 

(1) The high-leverage prototype AfcR candidates examined exhibit high degrees of uncertainty 
based in part, on schedules and deliverables. Lack of detailed proposal data is likely to 
have had some, albeit non-quantifiable, effect. 

(2) The use of probability distributions is helpful for aggregating and quantifying the 
magnitude of uncertainties. 

(3) The use of argument analysis, although time-consuming, proved useful for expressing the 
underlying patterns of reasoning for the numerical estimates of the attribute states. The 
numerical estimates provided a measure of the uncertainty whereas the argument analysis 
provided a window for the credibility of the estimates. The process used would be aided 
considerably by a computer-based interactive system. 

(4) There is a high degree of concordance (agreement) in the rankings at both the individual 
and group levels. There is similar robustness under different decision model assumptions. 
This is due to the comparable weightings assigned to the evaluation attributes by the 
interested parties interviewed. 

4. Recommendations 

(1) A high-leverage prototyping program should require the establishment of common goals 
and perspectives prior to the call for concepts. An element of setting such goals would be 
the development of program criteria to facilitate the collection of more complete 
information using a common format. 
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(2) If a high-leverage prototyping program is initiated, a formal call for proposals should be 
accompanied by a prescribed set of programmatic AfcR evaluation attributes and a 
common response format, to the extent feasible. 

(3) Establish an on-going AkR function at the systems engineering and integration level to 
support programmatic evaluations and systems engineering trade studies for the 

S.S. Freedom Program. Such a function has never been formally established within the 
S.S. Freedom Program. 

(4) Develop and implement a concrete plan for A&sR/new technology implementation and 
incorporation within the S.S. Freedom Program and constitute the plan into an operational 
process. 
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A&R 

AAWG 

AIESTWG 

Attribute 

Code ST 
DDT&E 

DMS 

ES 

FTS 

High-leverage prototype 
A&R candidate 

JEM-RMS 

KA 

KBS 

OMS 

SSRMS 

SPDM 

RWG 


i 

Automation and Robotics 
Advanced Automation Working Group 

Artificial Intelligence and Expert Systems Technology Working Group 

Measurable quantity used to establish the performance and value of a high- 
leverage prototype A&R candidate 

TBD 

Design, Development, Testing, and Evaluation Costs (Includes 
software costs) 

Data Management System 

Expert System 

Flight Telerobotic Servicer 


Type of A&R candidate with high potential for contribution to baseline S.S. 
Freedom. Leveraging refers to expectation that modest investment in the 
prototype will lead to early benefits in baseline S.S. Freedom design. 

Japanese Experiment Module— Remote Manipulator System 

Knowledge Acquisition 

Knowledge- Based System 

Operations Management System 

Space Station Remote Manipulator System 

Special-Purpose Dextrous Manipulator 

Robotics Working Group 

Group of contractors constructing elements of S.S. Freedom . There are four 
work packages 


Work Package 
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NASA LEVEL II AUTOMATION AND ROBOTICS 
CANDIDATE EVALUATION: 

HIGH-LEVERAGE PROTOTYPES 

QUESTIONNAIRE FOR OBTAINING PREFERENCE INFORMATION 
FOR USE IN RANKING A&R PROPOSAL CANDIDATES 

INTERVIEWEE: 

AFFILIATION: 

LOCATION: 

DATE: 

INTERVIEW #: 
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PURPOSE OF THE INTERVIEW 


THE PURPOSE OF THE INTERVIEW IN WHICH YOU ARE BEING 
ASKED TO PARTICIPATE IS TO COLLECT THE JUDGMENTS 
OF LEVEL II DECISION MAKERS REGARDING IMPORTANT 
FACTORS TO CONSIDER IN RANKING A<5cR PROPOSALS. 

THE QUESTIONS ARE AIMED AT OBTAINING YOUR PREFERENCES 
FOR SEVERAL FACTORS (E.G., COST. PRODUCTIVITY, ETC.) 
PERTINENT TO THE IDENTIFICATION OF PROMISING CONCEPTS 
FOR FURTHER DEVELOPMENT. 

THE RESPONSE OF ALL PERSONS INTERVIEWED WILL BE 
INCORPORATED IN THE RANKING OF A&R CANDIDATES. 

THE INTERVIEW IS DESIGNED TO TAKE 60 MINUTES. 
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AUTOMATION AND ROBOTICS CANDIDATES: 
HIGH-LEVERAGE PROTOTYPES 


THE AScR PROPOSAL CANDIDATES SPAN A VARIETY OF SPACE 
STATION FREEDOM TECHNOLOGIES, SUBSYSTEMS, DESIGN 
ENVIRONMENTS AND WORK AREAS ACROSS BOTH HARDWARE 
AND SOFTWARE 

THE EVALUATION CONSISTS OF 21 CANDIDATES. 
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SPACE STATION FREEDOM REQUIREMENTS 


LOW INVESTMENT COST 
LOW OPERATIONS COSTS 
MAXIMIZE CREW PRODUCTIVITY 
MAXIMIZE SAFETY 
MINIMIZE DEVELOPMENT RISK 
MINIMIZE RESOURCE CONSTRAINTS 
MAXIMIZE GROWTH POTENTIAL 
MAXIMIZE SPINOFF POTENTIAL 
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ATTRIBUTES AND THEIR RANGES 

RANGE 

ATTRIBUTE WORST CASE BEST CASE 


INITIAL (DDTE) COST SAVINGS 

0 

1 o 

OPERATIONS COST 

0 

1 o 

CREW PRODUCTIVITY 

O 

1 o 

SAFETY 

0 

1 o 

DEVELOPMENT RISK 

0 

1 0 

RESOURCE REQUIREMENTS 

0 

1 o 

GROWTH POTENTIAL 

0 

1 0 

SPINOFF POTENTIAL 

0 

1 o 
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Attribute Scale for Initial Cost Impacts 


Value 

Scale 

Rating 

Attribute Description 

Best Case 

10 

Funding of this proposal will substantially reduce DDT&E and system 
integration costs for the Phase I SSF. Very large net initial cost savings. 


8 

This proposal will reduce systems integration costs and potentially 
reduce DDT&E costs for the Phase I SSF. Initial cost savings from the 
investment are very likely. 


6 

This proposal will have minimal DDT&E cost savings for the Phase I 
SSF. Impacts are likely during the operations/evolution phase. 


4 

Proposal supports SSF assembly but DDTfeE costs could be significant. 


2 

Proposal supports SSF assembly but DDT&E costs could be large. 
Some issues for systems integration. 

Worst Case 

0 

Proposal provides minimal SSF benefits at high cost. Significant 
system integration costs. 
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Attribute Scale for Operations Cost Impacts 


Value and On-Orbit Costs 

Scale Rating Description 

Ground Operations 
Costs Description 

Crew Training 
Costs Description 

Best 

Case 

10 

Large expected O&M savings due 
to implementation of proposal 
deliverables. Savings are more 
than A&R alone; cross-cutting 
technology savings are likely. 
Savings extend across SSF 
lifetime 

Large O&M savings due to 
proposal are robust-savings 
impacts will be obtained for grnd 
ops. across SSF lifetime. 

Large O&M savings on 
costs due to implementation 
of proposal deliverables are 
likely. 

7 

O&M savings are focused with- 
in specific subsystems. Savings 
extend across SSF lifetime. 

Significant grnd. O&M savings 
across SSF lifetime sufficient 
to break even against DDT&E 
cost. 

Significant O&M savings on 
training costs. 

5 

O&M savings are likely and 
primary savings are expected 
during the 1st half of the SSF 
lifetime. 

Grnd. O&M savings are 
sufficient to break even when 
weighed against initial invest- 
ment cost (DDT&E). Savings 
will occur during 1st half of 
SSF lifetime. 

Some O&M savings in crew 
training costs. 

3 

O&M savings are expected 
during the 2nd half of the SSF 
lifetime. 

Grnd, O&M savings are 
uncertain and likely to occur 
during the 2nd half of the SSF 
lifetime. 

The crew training benefits 
for the proposal are 
uncertain. 

0 

Worst 

Case 

O&M savings are likely to be 
negative (no savings). 

Grnd. O&M savings are likely 
to be negative. 

Negative O&M savings on 
crew training costs. 
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Attribute Scale for Crew Productivity Impacts 


Value 

Scale 

Rating 

On-Orbit Productivity 
Description 

Ground Productivity 
Description 

Best Case 

10 

Very large improvements in crew 
productivity are likely if the proposal 
deliverables are implemented. 

Large improvements in ground crew 
productivity are likely. 


7 

Significant Improvements in crew 
productivity are likely. 

Significant improvements in ground 
crew productivity are likely. 


5 

Some improvements in crew 
productivity are likely. 

Some improvements in ground crew 
productivity are likely. 


3 

Few improvements in crew 
productivity are likely if the 
proposal deliverables are 
Implemented. 

Few improvements in ground crew 
productivity are likely. 

Worst Case 

0 

Little if no improvements in crew 
productivity are likely. 

No improvements in ground crew 
productivity. 
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Attribute Scale for Safety 


Value and On-Orbit Safety 

Scale Rating Description 

Best Case Very large improvements in crew safety Large improvements in ground safety are 

10 are likely if the proposal deliverables are likely, 

implemented. Adds safety benefits be- 
yond those achievable with man-in-the- 
loop due to high speed computing and/or 
reduction in EVA. Hazards can be 
mitigated by diagnosis and transition of 
system to automated n safe-hold" mode. 

8 Moderate safety improvement is likely. Moderate improvements in ground safety are 

Proposal technologies will increase safety likely, 

via moderate reductions in required EVA. 

6 Some improvements in safety. Some improvements in ground safety are 

Implementation of proposal deliverables likely, 

likely to increase safety by preventing 
improper or ill-defined command sequences 
from entering the control system. No major 
effects on EVA activity. 

4 Minor improvement in safety. Addition of Minor Improvements in ground safety are 

proposal deliverables increases safety in likely, 

specific areas by monitoring critical 
functions and data reduction of SSF status 
to smaller numbers of parameters. No 
expected effects on EVA. 

2 Little improvement in safety. Addition of Little, if any improvement in ground safety is 

proposal technology may be for reasons likely, 

other than safety but benefits may be 
accrued to safety in specific (and perhaps 
unlikely circumstances). There may be a 
small increase in required EVA. 

0 No improvements in safety above minimum No improvements in ground safety above 

Worst Case requirements. minimum requirements. 


Ground Safety 
Description 
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Attribute Scale for Development Risk 


Value 

Scale 

Rating 

Attribute Description 

Best Case 

10 

Virtually no development risk; off-the-shelf technologies; 
primarily modifications to the current system— mainly software; 
minimal hardware modifications; no testbed facilities required; 
low cost. 


7 

Some development risk; testing required for minor interfaces 
and verification purposes; existing testbeds can be modified 
and used; software and hardware modifications required; 
demonstrated feasibility; low level cost uncertainties. 


5 

Moderate risk development; some elements may require 
experiments and ground testing; some new testbed facilities 
are required; demonstrated feasibility; moderate cost. 


3 

High-risk development; numerous elements require ground 
testing; new testbeds need to be constructed; some in-space 
testing; feasibility relatively certain. 

Worst Case 

0 

Very high-risk development; elements at this level not demon- 
strated; may not be feasible. Requires new testbed concepts 
be constructed. 
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Attribute Scale for Resource Requirements Impacts 


Value and 


Non- Consumables 

Consumables 

Scale Rating 
— 

Power 

Upmass 

Upmass 


Best 

Case 

10 

The proposal implementation 
could result in large power 
savings. 

Large savings in non-con- 
sumables Is likely. 

Large savings in consum- 
ables is likely. 

8 

The proposal could result In 
significant power savings. 

Moderate savings in non-con- 
sumables is likely. 

Moderate savings in con- 
sumables is likely. 

6 

The proposal could result in 
some power savings. 

Some savings in non-con- 
sumables is likely. 

Some savings in con- 
sumables is likely. 

4 

The proposal could 

yield minor savings in power. 

Minor savings in non-con- 
sumables is possible. 

Minor savings in con- 
sumables is possible. 

2 

The proposal would have 
little impact on power reqts. 

Little, if any savings in non- 
consumables. 

Little, if any savings in 
consumables required. 

0 

Worst 

Case 

The proposal could increase 
power consumption. 

Could increase non-consumable 
upmass requirements. 

Could increase consumables 
upmass requirements. 
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Attribute Scale for Growth Potential 


Scale 

Value Rating Attribute Description 

Best Case 10 Proposal would result in a large net improvement in SSF’s 

ability to evolve or accept hooks and scars. 

7 Proposal would result in a moderate net improvement in SSF’s 

ability to evolve or accept later upgrades. 

5 Proposal would result in no net change in SSF’s ability to 

evolve or accept later upgrades. 

3 Proposal would result in minor negative net impacts on SSF’s 

ability to evolve and accept later upgrades. 

Worst Case 0 Proposal would result in some negative net impacts on SSF's 

ability to evolve and accept later upgrades. 
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Attribute Scale for Spinoff Potential 


Value 

Scale 

Rating 

Attribute Description 

Best Case 

10 

Proposal would have very high level of application to medium 
and low technology areas. Technologies will permeate into 
large markets (e.g., households). 


7 

Proposal would have many applications. Automation activities 
are applicable to medium technology areas and usable for many 
areas of manufacturing and production. 


5 

Proposal would yield some terrestrial applications. Main 
terrestrial examples are spinoffs to high technology applications 
such as computer software and manufacturing. 


3 

Few terrestrial applications. Proposal developments are aimed 
at aerospace and military applications where costs are high 
and technology-specific (e.g., automated navigation and attitude 
control). Main terrestrial applications are spinoffs from SSF 
to other space applications such as unmanned satellites. 

Worst Case 

0 

Little, if no terrestrial applications. All developments are 
SSF-specific and yield minimal technology transfer. Primary 
sources for transfer are documents describing developments 
and techniques. Few generalized methods or approaches to 
automation and robotics. 
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ATTRIBUTE: operations cost impacts 

GAMBLE SURE THING 
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ATTRIBUTE: crew productivity 

GAMBLE SURE THING 
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ATTRIBUTE: CREW PRODUCTIVITY 
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INDIFFERENCE POINT 




ATTRIBUTE: SAFETY 
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ATTRIBUTE: safety 
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INDIFFERENCE 









ATTRIBUTE: DEVELOPMENT 





ATTRIBUTE: development 












ATTRIBUTE: RESOURCE REQUIREMENTS 

GAMBLE SURE THING 
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IF YOU KNEW THAT ALL OTHER ATTRIBUTES WERE 
AT THEIR BEST STATES? 
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ATTRIBUTE: GROWTH POTENTIAL 
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ATTRIBUTE: SPINOFF POTENTIAL 






















































IMPORTANCE OF: INITIAL 
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WHAT DO YOU WIN IF YOU WIN THE GAMBLE? WHAT DO YOU LOSE IF YOU LOSE THE GAMBLE? 





IMPORTANCE OF: OPERATIONS COST 






IMPORTANCE OF: CREW PRODUCTIVITY 
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WHAT DO YOU WIN IF YOU WIN THE GAMBLE? WHAT DO YOU LOSE IE YOU LOSE THE GAMBLE? 





IMPORTANCE OF: 



WHlAT DO YOU WIN IF YOU WIN THE GAMBLE? 
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WHAT DO YOU WIN IF YOU WIN THE GAMBLE? WHAT DO YOU LOSE IF YOU LOSE THE GAMBLE? 






WHAT DO YOU WIN IF YOU WIN THE GAMBLE? 





IMPORTANCE OF: GROWTH POTENTIAL 


Appendix B: Interview Questionnaire for Preference Assessment 


UJ 

m 

< 

o 


X x X X X x 

oooooo 

O CO 0 1 w 




5 

X 

5 


cr 

o 

Ll 



Ll 

UJ 

cr 


C" 

I 


< 


C 

c 

c 

h- 

< 

$ 


WHAT DO YOU WIN IF YOU WIN THF GAMBI.F? 





IMPORTANCE OF: SPINOFF 



B - 48 


WHAT DO YOU WIN IF YOU WIN THE GAMBLE? WHAT DO YOU LOSE IE YOU UOSE Tl IE GAMBl F? 
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APPENDIX C 

DECISION MODEL CASES AND 
SAMPLE COMPUTER OUTPUT FOR SENSITIVITY ANALYSIS 


1 
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Appendix C: Sample Computer Output for Nominal Case Analysis 


Table C-l. 


Decision Model Sensitivity Analysis Cases 1 


OUTPUT DECISION MODEL CASE 

FILENAME OPTIONS NUMBER 


Original Baseline Disk Input File 



HLEV000 

DET/A/NOM/PRIM/MULT 

0 

Deterministic Baseline-Nominal Case 



HLEV001 Group 1 

DET/l/NOM/PRIM/MULT 

1 

HLEV002 Group 2 

DET/2/NOM/PRIM/MULT 

2 

HLEV003 Group 3 

DET/3/NOM/PRIM/MULT 

3 

HLEV004 Group All 

DET/A/NOM/PRIM/MULT 

4 

Probabilistic Baseline-Nominal Case 



HLEV011 Group 1 

PROB/l/NOM/PRIM/MULT 

5 

HLEV012 Group 2 

PROB/2/NOM/PRIM/MULT 

6 

HLEV013 Group 3 

PROB/3/NOM/PRIM/MULT 

7 

HLEV014 Group All 

PROB/A/NOM/PRIM/MULT 

8 

Deterministic Baseline-Best Case 



HLEV051 Group 1 

DET/l/BEST/PRIM/MULT 

9 

HLEV052 Group 2 

DET/2/BEST/PRIM/MULT 

10 

HLEV053 Group 3 

DET/3/BEST/P RIM/MULT 

11 

HLEV054 Group All 

DET/A/BEST/PRIM/MULT 

12 

Deterministic Baseline-Worst Case 



HLEV061 Group 1 

DET/l/WORST/PRIM/MULT 

13 

HLEV062 Group 2 

DET/2/WORST/PRIM/MULT 

14 

HLEV063 Group 3 

DET/3/WORST/PRIM/MULT 

15 

HLEV064 Group All 

DET/A/WORST/PRIM/MULT 

16 


1 The potential number of cases to be examined is: 

2: Deterministic (DET); Probabilistic (PROB). 

4: groups composed of interested parties 1,2,3, and (A)LL, 

3: Nominal states (NOM), Best Case attribute states, 

(BEST), and Worst Case attribute states (WORST). 

1: One primary data set analyzed (PRIM) 

x 2: Multiplicative Model (MULT); Additive Model (ADD). 

= 48 combinations 
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Analysis Mode 

Number of cases 

Attribute state assumptions 

Attribute data estimates 
Decision models 
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Table C-l. Decision Model Sensitivity Analysis Cases (Continued) 


OUTPUT 

FILENAME 

DECISION MODEL CASE 

OPTIONS NUMB 

J 

ER 

Probabilistic Baseline-Best Case 

PROB/l/BEST/PRIM/MULT 

17 

HLEV021 Group 1 

HLEV022 Group 2 

PROB/2/BEST/PRIM/MULT 

18 

HLEV023 Group 3 

PROB/3/BEST/PRIM/MULT 

19 

HLEV024 Group All 

PROB/A/BEST/PRIM/MULT 

20 

Probabilistic Baseline-Worst Case 


21 

HLEV031 Group 1 

PROB/l/WORST/PRIM/MULT 

HLEV032 Group 2 

PROB/2/WORST/PRIM/MULT 

22 

HLEV033 Group 3 

PROB/3/WORST/PRIM/MULT 

23 

HLEV034 Group All 

PROB/A/WORST/PRIM/MULT 

24 

Deterministic Baseline-Nominal Case 
Checks differences between decision models 


25 

HLEVlll Group 1 

DET/l/NOM/PRIM/ADD 

HLEV112 Group 2 

DET/2/NOM/PRIM/ADD 

26 

HLEV113 Group 3 

DET/3/NQM/PRIM/ADD 

27 

HLEV114 Group All 

DET/A/NOM/PRIM/ADD 

28 

Probabilistic Baseline-Nominal Case 
Checks differences between decision models 


29 

HLEV041 Group 1 

PROB/l/NOM/PRIM/ADD 

HLEV042 Group 2 

PROB/2/NOM/PRIM/ADD 

30 

HLEV043 Group 3 

PROB/3/NOM/PRIM/ADD 

31 

HLEV044 Group All 

PROB/A/NOM/PRIM/ADD 

32 

Deterministic Baseline- Best Case 
Checks differences between decision models 


33 

HLEV071 Group 1 

DET/l/BEST/PRIM/ADD 

HLEV072 Group 2 

DET/2/BEST/PRIM/ADD 

34 

HLEV073 Group 3 

DET/3/BEST/PRIM/ADD 

35 

HLEV074 Group All 

DET/A/BEST/PRIM/ADD 

36 

Deterministic Baseline- Worst Case 
Checks differences between decision models 


37 

HLEV081 Group 1 

DET/l/WORST/PRIM/ADD 

HLEV082 Group 2 

DET/2/WORST/PRIM/ADD 

38 

HLEV083 Group 3 

DET/3/WORST/PRIM/ADD 

39 

HLEV084 Group All 

DET/A/WORST/PRIM/ADD 

40 
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Table C-l. Decision Model Sensitivity Analysis Cases (Continued) 


OUTPUT 

FILENAME 

DECISION MODEL CASE 

OPTIONS NUMBER 

Probabilistic Baseline-Best Case 



HLEV091 Group 1 

PROB/l/BEST/PRIM/ADD 

41 

HLEV092 Group 2 

PROB/2/BEST/PRIM/ADD 

42 

HLEV093 Group 3 

PROB/3/BEST/PRIM/ADD 

43 

HLEV094 Group All 

PROB/ A/BEST/PRIM /ADD 

44 

Probabilistic Baseline-Worst Case 
Checks differences between decision models 



HLEV101 Group 1 

PROB/l/WORST/PRIM/ADD 

45 

HLEV102 Group 2 

PROB/2/ WORST/PRIM/ A DD 

46 

HLEV103 Group 3 

PROB/3/WORST/PRIM/ADD 

47 

H LEV 104 Group All 

PROB/ A/WORST/PRIM/ADD 

48 









SUMMARY REPORT 
FOR CASE: HLEVOOO 


This report sunmartzes the results of HLEVOOO as follows 

I Introduction and Case Specifications 

A. The Alternatives 

B. The Attributes 

C. The Decision Makers 

D. The Analysis Mode 

E. The Run Option 

F. The Decision Model 

II Individual Decision Maker Rankings 

III Decision Maker Rankings As a Group 

IV Appendices 

A. The Attribute State/Distribution Inputs 

B. The Preference Inputs for Each Decision Maker 
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SECTION I 

INTRODUCTION AND CASE SPECIFICATIONS 

This Section presents the specifications for the decision making 
case defined. 

A. The Alternatives are: 

Alternative 1 = FAULTPRED 
Alternative 2 = PROTOSCHED 
Alternative 3 = EXTFAULTTB 
Alternative 4 = KBSFAULTRE 
Alternative 5 = VDDTOHADHS 
Alternative 6 = ACCELSSMPM 
Alternative 7 = ASRPROTOTY 
Alternative 8 = DIAGES/EPS 
Alternative 9 = ABDUCT /EPS 
Alternative 10 = STD/TLS/ES 
Alternative 11 = AUTROBASSY 
Alternative 12 = TRTHERMPOS 
Alternative 13 = EVARETRIEV 
Alternative 14 = AUTROBSIM 
Alternative 15 = STEREOTELE 
Alternative 16 = UORLDMOOEL 
Alternative 17 = ROBCTLASSY 
Alternative 18 = KNOWACOSTD 
Alternative 19 = ADAEFFECTI 
Alternative 20 = ES/STDSRES 
Alternative 21 = KBSDVMAINT 
Alternative 22 = DUMMY 

B. The Attributes are: 

Attribute 1 = INITIAL COST IMPCT 
Attribute 2 » OPERATIONS COSTS 
Attribute 3 = CREW PRODUCTIVITY 
Attribute 4 = SAFETY 

Attribute 5 = DEVELOPMENT RISK 
Attribute 6 = RESOURCE REQTS IMP 
Attribute 7 = GROWTH POTENTIAL 
Attribute 8 = SPINOFF POTENTIAL 

C. The Decision Makers are: 

Individual 1 = INTRVW 01 Individual 6 = INTRVU 06 

Individual 2 = INTRVW 02 Individual 7 = INTRVW 07 

Individual 3 = INTRVW 03 Individual 8 = INTRVU 08 

Individual 4 « INTRVU 04 Individual 9 = INTRVU 09 

Individual 5 * INTRVU 05 

D. Mode of Analysis Selected is: Detailed Analysis Mode 


E. Run Option for Uncertainty is: Probabilistic 


F. Type of Decision Model is: Detailed Non-Linear (Multiplicative) Model 
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SECTION II 

INDIVIDUAL DECISION MAKER RANKINGS 


This Section presents the results of a multi -attribute decision 
analysis for the Case HLEV000 with rankings for each decision maker. 

Rankings for Decision Maker: INTRVW 01 


Alternative 

Value 

Rank 

FAULTPRED 

0.9389 

10 

♦/- 

0.0103 


PROTOSCHED 

0.9520 

4 

♦/- 

0.0080 


EXTFAULTTB 

0.9434 

7 

+/- 

0.0168 


KBSFAULTRE 

0.9627 

3 

+/- 

0.0075 


VDDTOMADMS 

0.8666 

20 

♦/- 

0.0385 


ACCELSSMPM 

0.8898 

18 

♦/- 

0.0186 


A&RPROTOTY 

0.8719 

19 

♦/- 

0.0205 


DIAGES/EPS 

0.9747 

1 


0.0054 


ABDUCT/EPS 

0.9114 

13 

♦/- 

0.0171 


STD/TLS/ES 

0.9628 

2 

♦/- 

0.0093 


AUTROBASSY 

0.9422 

9 


0.0089 


TRTHERMPOS 

0.9145 

12 


0.0101 


EVARETRIEV 

0.9200 

11 

♦/- 

0.0178 


AUTROBSIM 

0.9092 

16 

♦/- 

0.0117 


STEREOTEIE 

0.8983 

17 

♦/- 

0.0179 


WORLDMOOEL 

0.8460 

21 

♦/- 

0.0464 


ROBCTIASSY 

0.9509 

5 

♦/- 

0.0073 


KNOWACQSTD 

0.9098 

15 


0.0143 


ADAEFFECTI 

0.9484 

6 

+/- 

0.0115 


ES/STDSRES 

0.9114 

14 

♦/- 

0.0214 


KBSDVMAINT 

0.9430 

8 

♦/- 

0.0164 


DUMMY 

0.4022 

22 

+/■ 

0.2463 
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Rankings for Decision Maker: IMTRVW 02 


Alternative Value 

Rank 

FAUITPRED 0.7858 
+/- 0.0300 

8 

PROTOSCHED 0.7442 
♦/- 0.0283 

12 

EXTFAULTTB 0.7892 
♦/- 0.0291 

7 

KBSFAULTRE 0.8452 
♦/- 0.0255 

2 

VDDTOMADMS 0.6689 
♦/- 0.0362 

16 

ACCELSSMPM 0.6327 
+/- 0.0310 

21 

A&RPROTOTY 0.6425 
♦/- 0.0275 

19 

DIAGES/EPS 0.8165 
+/- 0.0223 

4 

ABDUCT/EPS 0.6875 
♦/- 0.0118 

14 

STD/TLS/ES 0.8505 
+/- 0.0254 

1 

AUTROBASSY 0.7951 
+/- 0.0307 

6 

TRTHERMPOS 0.7064 
♦/- 0.0199 

13 

EVARETRIEV 0.6811 
♦/- 0.0307 

15 

AUTROBSIM 0.6592 
+/- 0.0203 

17 

STEREOTELE 0.6352 
+/- 0.0439 

20 

UORlDMOOEl 0.6439 
+/- 0.0348 

18 

ROBCTLASSY 0.7841 
♦/- 0.0219 

9 

ICMOUACQSTD 0.7983 
+/- 0.0288 

5 

ADAEFFECTI 0.8190 
♦/- 0.0303 

3 

ES/STDSRES 0.7595 
+/- 0.0494 

11 

KBSDVMAINT 0.7764 
+/- 0.0345 

10 

DUMMY 0.2413 
V- 0.1891 

22 
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Rankings for Decision Maker: INTRVW 03 


Alternative Value 

Rank 

FAULTPRED 0.7304 
♦/- 0.0195 

10 

PROTOSCHED 0.7715 
+/- 0.0163 

4 

EXTFAULTTB 0.7604 
+/- 0.0379 

6 

KBSFAULTRE 0.7842 
♦/- 0.0161 

2 

VDDTOMADMS 0.5941 
♦/- 0.0434 

21 

ACCELSSMPM 0.6618 
♦/- 0.0232 

19 

A&RPROTOTY 0.6928 
♦/- 0.0303 

13 

D1 AGES/EPS 0.8080 
♦/- 0.0134 

1 

ABDUCT/EPS 0.6945 
+/- 0.0192 

12 

STD/TLS/ES 0.7791 
*/- 0.0192 

3 

AUTROBASSY 0.7378 
+/- 0.0183 

9 

TRTHERHPOS 0.6825 
+/- 0.0156 

16 

EVARETRIEV 0.7197 
+/* 0.0238 

11 

AUTROBSIM 0.6763 
+/- 0.0149 

17 

STEREOTELE 0.6871 
*/- 0.0230 

15 

UORLDMOOEL 0.6003 
+/• 0.0543 

20 

ROBCTLASSY 0.7588 
♦/- 0.0160 

7 

KNOUACOSTD 0.6698 
♦/* 0.0210 

18 

ADAEFFECTI 0.7661 
*/- 0.0212 

5 

ES/STDSRES 0.6890 
+/- 0.0342 

14 

KBSDVMAINT 0.7428 
♦/- 0.0302 

8 

DUMMY 0.2411 
+/- 0.1647 

22 
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Rankings for Decision Maker: INTRW 04 


Alternative Value 

Rank 

FAULTPRED 0.9389 
♦/- 0.0103 

10 

PROTOSCHED 0.9520 
V- 0.0074 

6 

EXTFAULTTB 0.9495 
+/- 0.0107 

7 

KBSFAULTRE 0.9582 
+/- 0.0066 

3 

VDDTOMADMS 0.8614 
0.0305 

20 

ACCEISSMPM 0.8990 
+/“ 0.0158 

18 

A&RPROTOTY 0.9007 
+/- 0.0174 

17 

D I AGES/EPS 0.9727 
+/- 0.0046 

1 

ABOUCT/EPS 0.9012 
+/“ 0.0112 

16 

STD/TLS/ES 0.9618 
+/- 0.0078 

2 

AUTROBASSY 0.9435 
+/- 0.0076 

9 

TRTHERMPOS 0,9210 
+/- 0.0095 

11 

EVARETRIEV 0,9208 
+/- 0.0145 

12 

AUTROBSIM 0.9023 
+/- 0.0105 

15 

STEREOTEIE 0.8911 
+/- 0.0152 

19 

UORLDMODEl 0.8325 
♦/- 0.0359 

21 

ROBCTLASSY 0.9552 
♦/- 0.0059 

4 

KNOWACQSTO 0.9034 
♦/- 0.0141 

14 

AOAEFFECTI 0.9490 
♦/- 0.0102 

8 

ES/STDSRES 0.9106 
♦/• 0.0182 

13 

KBSOVMAINT 0.9539 
+/• 0.0094 

5 

DUMMY 0.4109 
+/- 0,2479 

22 
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Rankings for Decision Maker: INTRVW 05 


Alternative Value 

Rank 

FAULTPRED 0.9367 
+/- 0.0078 

10 

PROTOSCHED 0.9596 
+/- 0.0050 

4 

EXTFAULTTB 0.9503 
+/- 0.0153 

7 

KBSFAULTRE 0.9677 
*/- 0.0058 

3 

VDDTOMADMS 0.8649 
*/- 0.0357 

20 

ACCELSSMPM 0.9077 
+/- 0.0133 

18 

A4RPROTOTY 0.8923 
+/- 0.0142 

19 

D I AGES/EPS 0.9735 
+/- 0.0046 

1 

ABDUCT/EPS 0.9159 
♦/- 0.0168 

15 

STO/TLS/ES 0.9694 
+/- 0.0058 

2 

AUTROBASSY 0.9428 
+/- 0.0054 

9 

TRTHERMPOS 0.9264 
+/- 0.0076 

12 

EVARETRIEV 0.9351 
♦/- 0.0128 

11 

AUTROBSIM 0.9229 
*/• 0.0082 

13 

STEREOTELE 0.9081 
♦/- 0.0129 

17 

UORLDMOOEL 0.8607 
♦/- 0.0417 

21 

ROBOT LASSY 0.9566 
+/- 0.0046 

5 

KNOWACQSTD 0.9167 
+/- 0.0076 

14 

ADAEFFECTI 0.9563 
*/■ 0.0066 

6 

ES/STDSRES 0.9112 
+/- 0.0150 

16 

KBSDVHAINT 0.9447 
+/- 0.0137 

8 

DUMMY 0.4129 
+/- 0.2542 

22 


C - 11 


Rankings for Decision Maker: INTRVW 06 


Alternative Value 

Rank 

FAULTPRED 0.9308 
+/- 0.0148 

10 

PROTOSCHED 0.9473 
+/- 0.0101 

4 

EXTFAULTTB 0.9382 
+/- 0.0201 

9 

KBSFAUITRE 0.9565 
+/* 0.0100 

3 

VDDTOMADMS 0.8233 
+/- 0.0465 

20 

ACCEL SSWPM 0.8768 
+/- 0.0176 

18 

A&RPROTOTY 0.8710 
V- 0.0203 

19 

DIAGES/EPS 0.9696 
+/- 0.0082 

1 

ABDUCT/EPS 0.9060 
+/“ 0.0160 

13 

STD/TLS/ES 0.9591 
♦/- 0.0109 

2 

AUTROBASSY 0.9391 
+/- 0.0141 

8 

TRTHERMPOS 0.8882 
+/- 0.0112 

15 

EVARETRIEV 0.9077 
+/- 0.0201 

11 

AUTROBSIM 0.8872 
+/- 0.0134 

16 

STEREOTELE 0.8957 
+/- 0.0238 

14 

UORLDMODEL 0.8042 
+/- 0.0574 

21 

ROBCTLASSY 0.9449 
+/• 0.0096 

5 

KNOWACGSTD 0.8861 
+/- 0.0235 

17 

ADAEFFECTI 0.9424 
♦/- 0.0147 

6 

ES/STDSRES 0.9075 
+/- 0.0292 

12 

KBSDVMAINT 0.9400 
V 0.0187 

7 

DUMMY 0.4134 
+/‘ 0.2620 

22 
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Rankings for Decision Maker: IMTRVU 07 


Alternative Value 

Rank 

FAULTPRED 0.7083 
+/- 0.0198 

10 

PROTOSCHED 0.7227 
+/- 0.0180 

9 

EXTFAULTTB 0.7445 
♦/- 0.0388 

6 

KBSFAULTRE 0.7496 
*/- 0.0213 

5 

VDDTOMADHS 0.5594 
+/- 0.0527 

20 

ACCELSSMPM 0.6609 
♦/- 0.0202 

14 

A&RPROTOTY 0.5708 
♦/- 0.0265 

19 

DIAGES/EPS 0.7902 
+/- 0.0216 

1 

ABOUCT/EPS 0.6459 
♦/- 0.0327 

16 

STD/TLS/ES 0.7832 
♦/- 0.0229 

2 

AUTROBASSY 0.7435 
+/- 0.0175 

7 

TRTHERMPOS 0.6827 
+/- 0.0140 

11 

EVARETRIEV 0.6745 
+/- 0.0273 

12 

AUTROBSIM 0.6557 
+/~ 0.0191 

15 

STEREOTELE 0.6363 
+/- 0.0262 

17 

WORLDMOOEl 0.5357 
+/- 0.0560 

21 

ROBOT L ASSY 0.7511 
+/- 0.0157 

3 

KNOWACQSTD 0.6268 
♦/- 0.0187 

18 

AOAEFFECTI 0.7339 
♦/- 0.0215 

8 

ES/STDSRES 0.6640 
+/- 0.0292 

13 

KBSDVMAINT 0.7496 
+/- 0.0364 

4 

DUMMY 0.2112 
+/- 0.1519 

22 
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Rankings for Decision Maker: INTRVW 08 


Alternative Value 

Rank 

FAULTPRED 0.9143 
+/- 0.0109 

10 

PROTOSCHED 0.9263 
♦/- 0.0095 

7 

EXTFAULTTB 0.9303 
+/- 0.0162 

6 

KBSFAULTRE 0.9450 
+/- 0.0077 

3 

VDDTOMADMS 0.8150 
+/- 0.0357 

19 

ACCEL SSMPM 0.8665 
+/- 0.0173 

18 

A&RPROTOTY 0.8126 
+/- 0.0244 

20 

D I AGES/EPS 0.9534 
♦/- 0.0059 

1 

ABDUCT/EPS 0.8740 
+/- 0.0136 

16 

STD/TLS/ES 0.9491 
+/“ 0.0086 

2 

AUTROBASSY 0.9261 
♦/- 0.0087 

8 

TRTHERMPOS 0.8947 
+/- 0.0099 

11 

EVARETRIEV 0.8932 
♦/- 0.0160 

12 

AUTROBSIM 0.8688 
+/- 0.0114 

17 

STEREOTELE 0.8795 
+/- 0.0151 

15 

WORIDMOOEL 0.7992 
+/- 0.0441 

21 

ROBCTLASSY 0.9321 
♦/- 0.0075 

5 

KNOUACQSTD 0.8849 
♦/- 0.0138 

14 

ADAEFFECTI 0.9348 
♦/- 0.0112 

4 

ES/STDSRES 0.8854 
+/- 0.0211 

13 

KBSDVMAINT 0.9259 
+/- 0.0149 

9 

DUMMY 0.3772 
♦/- 0.2333 

22 
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Rankings for Decision Maker: INTRVW 09 


Alternative Value 

Rank 

FAULTPRED 0.9298 
+/- 0.0132 

10 

PROTOSCHED 0.9491 
+/- 0.0088 

5 

EXTFAULTTB 0.9434 
+/- 0.0184 

6 

KBSFAULTRE 0.9642 
+/- 0.0077 

3 

VDDTOMADMS 0.8196 
+/- 0.0441 

20 

ACCELSSMPM 0.8840 
+/- 0.0169 

18 

A&RPROTOTY 0.8575 
+/- 0.0251 

19 

DIAGES/EPS 0.9718 
+/- 0.0056 

1 

ABDUCT/EPS 0.9007 
+/- 0.0144 

16 

STD/TLS/ES 0.9654 
+/- 0.0077 

2 

AUTROBASSY 0.9347 
+/“ 0.0113 

9 

TRTHERMPOS 0.9036 
+/' 0.0116 

13 

EVARETRIEV 0.9196 
♦/- 0.0160 

11 

AUTROBSIM 0.8861 
+/- 0.0121 

17 

STEREOTELE 0.9128 
+/- 0.0165 

12 

WORLDMODEl 0.8176 
+/- 0.0554 

21 

ROBCTLASSY 0.9413 
♦/- 0.0086 

7 

KNOWACQSTD 0.9034 
0.0163 

14 

ADAEFFECTI 0.9525 
+/- 0.0110 

4 

ES/STDSRES 0.9022 
+/- 0.0229 

15 

KBSDVMAINT 0.9362 
+/- 0.0167 

8 

DUMMY 0.4199 
+/- 0.2622 

22 


The rankings are in Agreement at a 95 percent significance level. 
The value of the concordance (0= agreement, ^disagreement) is 0.93 
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SECTION III 

DECISION MAKER GROUP RANKINGS 

This Section presents the results for Case HLEV000 with rankings 
for all decision makers as a group using three different rules. 

Additive Rule * Sum of Scores 

Nash Bargaining Rule = Product of Scores 
Rank Sum Rule = Sum of Ranks 

Additive Nash Bargaining Rank Si m 


Alternative 

Rule 

Rule 

Rule 


Value 

Rank 

Value 

Rank 

Value 

Rank 

FAULTPRED 

0.6682 

10 

0.8631 

10 

0.5820 

10 

PROTOSCHED 

0.6805 

7 

0.8750 

7 

0.7566 

6 

EXTFAULTTB 

0.8832 

6 

0.8790 

6 

0.7249 

7 

KBSFAULTRE 

0.9037 

3 

0.8998 

3 

0.9048 

3 

VDDTOMADMS 

0.7637 

20 

0.7542 

20 

0.1164 

20 

ACCEL SSMPM 

0.8088 

18 

0.8005 

18 

0.1905 

18 

A&RPROTOTY 

0.7902 

19 

0.7809 

19 

0.1799 

19 

D I AGES/EPS 

0.9145 

1 

0.9110 

1 

0.9841 

1 

ABDUCT/EPS 

0.8264 

15 

0.8188 

15 

0.3545 

15 

STD/TLS/ES 

0.9089 

2 

0.9055 

2 

0.9524 

2 

AUTROBASSY 

0.8783 

9 

0.8739 

9 

0.6561 

9 

TRTHERMPOS 

0.8356 

13 

0.8288 

13 

0.4444 

12 

EVARETRIEV 

0.8413 

11 

0.8340 

11 

0.4868 

11 

AUTROBSIM 

0.8186 

16 

0.8107 

16 

0.2910 

16 

STEREOTELE 

0.8160 

17 

0.8070 

17 

0.2751 

17 

WORLD MOO EL 

0.7489 

21 

0.7393 

21 

0.0688 

21 

ROBCTLASSY 

0.8861 

5 

0.8817 

5 

0.7831 

4 

KNOUACQSTD 

0.8333 

14 

0.8260 

14 

0.3651 

14 

ADAEFFECTI 

0.8892 

4 

0.8849 

4 

0.7831 

4 

ES/STDSRES 

0.8379 

12 

0.8317 

12 

0.4074 

13 

KBSDVMAINT 

0.8792 

8 

0.8746 

8 

0.6931 

8 

DUMMY 

0.3478 

22 

0.3362 

22 

0.0000 

22 


The group rankings are in Agreement at a 95 percent significance level. 
The value of the concordance <0=agreement # 1=disagreement) is 1.00 
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SECTION IV 
DATA INPUTS 

This Section presents the inputs for Case HLEV000 used to compute 
the rankings. 

A. The Attribute State/Distribution Inputs 
Attribute States for ALTERNATIVE: FAULTPRED 


Point Value for INITIAL 

COST 

IMPCT 

6.0000 


Cunulative Distribution 

for 

OPERATIONS COSTS 



State CDF 


State CDF 

State 

CDF 

3.0000 0.00 


4.0000 0.50 

6.0000 

1.00 

3.7000 0.25 


4.8000 0.75 



Point Value for CREW PRODUCTIVITY 

5.0000 


Cumulative Distribution 

for 

SAFETY 



State CDF 


State CDF 

State 

CDF 

3.0000 0.00 


6.0000 0.50 

8.0000 

1.00 

4.8000 0.25 . 


6.8000 0.75 



Point Value for DEVELOPMENT 

RISK * 

5.0000 


Cumulative Distribution 

for 

RESOURCE REQTS IMP 



State CDF 


State CDF 

State 

CDF 

5.0000 0.00 


6.0000 0.50 

8.0000 

1.00 

5.9000 0.25 


7.2000 0.75 



Cumulative Distribution 

for 

GROWTH POTENTIAL 



State CDF 


State CDF 

State 

CDF 

6.0000 0.00 


7.0000 0.50 

8.0000 

1.00 

6.5000 0.25 


7.5000 0.75 



Cumulative Distribution 

for 

SPINOFF POTENTIAL 



State CDF 


State CDF 

State 

CDF 

0.0000 0.00 


1.5000 0.50 

3.0000 

1.00 

0.7500 0.25 


2.2500 0.75 




Attribute States for ALTERNATIVE: PROTOSCHED 

Point Value for INITIAL 

COST 

IMPCT 

5.0000 


Cumulative Distribution 

for 

OPERATIONS COSTS 



State CDF 


State CDF 

State 

CDF 

5.0000 0.00 


6.5000 0.50 

7.0000 

1.00 

5.4000 0.25 


6.5000 0.75 



Cumulative Distribution 

for 

CREW PRODUCTIVITY 



State CDF 


State CDF 

State 

CDF 

7.0000 0.00 


7.5000 0.50 

8.0000 

1.00 

7.2500 0.25 


7.7500 0.75 



Cumulative Distribution 

for 

SAFETY 



State CDF 


State CDF 

State 

CDF 

1.0000 0.00 


3.0000 0.50 

7.0000 

1.00 

2.1000 0.25 


4.2000 0.75 
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Point Value for DEVELOPMENT 
Cunulative Distribution for 
State CDF 

3.0000 0.00 

3.5000 0.25 


RISK 

RESOURCE REOTS IMP 
State CDF 
4.0000 0.50 
4.5000 0.75 


5.0000 

State CDF 
5.0000 1.00 


Cumulative Distribution for GROWTH POTENTIAL 


State CDF 

6.0000 0.00 

6.5000 0.25 


State CDF 

7.0000 0.50 

7.5000 0.75 


State CDF 

8.0000 1.00 


Cumulative Distribution for SPINOFF POTENTIAL 


State CDF 

6.0000 0.00 

6.7500 0.25 


State CDF 

7.5000 0.50 
8.2500 0.75 


State CDF 

9.0000 1.00 


Attribute States for ALTERNATIVE: EXTFAULTTB 


Point Value for INITIAL 

COST IMPCT 

7.0000 


Cumulative Distribution 

for 

OPERATIONS COSTS 



State CDF 


State CDF 

State 

CDF 

3.5000 0.00 


4.0000 0.50 

5 . 5000 

1.00 

3.9000 0.25 


4.9000 0.75 



Cunulative Distribution 

for 

CREW PRODUCTIVITY 



State CDF 


State CDF 

State 

CDF 

4.0000 0.00 


4.5000 0.50 

5.0000 

1.00 

4.2500 0.25 


4.7500 0.75 



Cumulative Distribution 

for 

SAFETY 



State CDF 


State CDF 

State 

CDF 

2.0000 0.00 


5.0000 0.50 

8.0000 

1.00 

3.8000 0.25 


6,3000 0.75 



Point Value for DEVELOPMENT 

RISK 

8.0000 


Cumulative Distribution 

for 

RESOURCE REQTS IMP 



State CDF 


State CDF 

State 

CDF 

5.0000 0.00 


6.0000 0.50 

6.8000 

1.00 

5.8000 0.25 


6.6000 0.75 



Cumulative Distribution 

for 

GROWTH POTENTIAL 



State CDF 


State CDF 

State 

CDF 

0.0000 0.00 


5.0000 0.50 

10.0000 

1.00 

2.5000 0.25 


7.5000 0.75 



Cumulative Distribution 

for 

SPINOFF POTENTIAL 



State CDF 


State CDF 

State 

CDF 

0.0000 0.00 


5.0000 0.50 

10.0000 

1.00 




Attribute States for ALTERNATIVE: KBSFAULTRE 


Cumulative Distribution 

for 

INITIAL COST IMPCT 



State CDF 


State CDF 

State 

CDF 

8.0000 0.00 


8.5000 0.50 

9.0000 

1.00 

8.2500 0.25 


8.7500 0.75 



Cumulative Distribution 

for 

OPERATIONS COSTS 



State CDF 


State CDF 

State 

CDF 

5.0000 0.00 


6.5000 0.50 

8.0000 

1.00 

6.4000 0.25 


7.2000 0.75 



Point Value for CREW PRODUCTIVITY 

6.0000 


Cumulative Distribution 

for 

SAFETY 



State CDF 


State CDF 

State 

CDF 

2.0000 0.00 


5.0000 0.50 

8.0000 

1.00 

3.8000 0.25 


6.4000 0.75 



Point Value for DEVELOPMENT 

RISK = 

4.0000 


Cumulative Distribution 

for 

RESOURCE REQTS IMP 



State CDF 


State CDF 

State 

CDF 

4.0000 0.00 


4.5000 0.50 

5.0000 

1.00 

4.2500 0.25 


4.7500 0.75 



Cumulative Distribution 

for 

GROWTH POTENTIAL 



State CDF 


State CDF 

State 

CDF 

5.0000 0.00 


7.0000 0.50 

9.0000 

1.00 

6.0000 0.25 


8.0000 0.75 



Cumulative Distribution 

for 

SPINOFF POTENTIAL 



State CDF 


State CDF 

State 

CDF 

5.0000 0.00 


6.0000 0.50 

7.0000 

1.00 

5.5000 0.25 


6.5000 0.75 




Attribute States for ALTERNATIVE: VDDTOHADMS 


Point Value for INITIAL COST IMPCT = 5.0000 
Cumulative Distribution for OPERATIONS COSTS 

State CDF State CDF State CDF 

3.0000 0.00 5.0000 0.50 6.5000 1.00 
4.5000 0.25 5.4000 0.75 


Cumulative Distribution 
State COF 

4.0000 0.00 
4.2500 0.25 


for CREW PRODUCTIVITY 
State CDF 
4.5000 0.50 
4.7500 0.75 


State CDF 
5.0000 1.00 


Cumulative Distribution for SAFETY 


State CDF 

1.0000 0.00 

2.0000 0.25 


State CDF 

3.0000 0.50 
4.3000 0.75 


State CDF 

6.0000 1.00 


Point Value for DEVELOPMENT 
Cumulative Distribution for 
State CDF 

1.0000 0.00 

2.3100 0.25 


RISK 

RESOURCE REGTS IMP 
State CDF 
2.5000 0.50 
2.7500 0.75 


2.0000 

State CDF 
3.0000 1.00 
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Cumulative Distribution for GROWTH POTENTIAL 

State COF State CDF State CDF 
0.0000 0.00 5.0000 0.50 10.0000 1.00 

2.5000 0.25 7.5000 0.75 

Cumulative Distribution for SPINOFF POTENTIAL 

State CDF State CDF State CDF 
0.0000 0.00 1.5000 0.50 3.0000 1.00 
0.7500 0.25 2.2500 0.75 


Attribute States for ALTERNATIVE: ACCELSSHPM 

Point Value for INITIAL COST IHPCT « 4.0000 

Cumulative Distribution for OPERATIONS COSTS 

State CDF State CDF State CDF 

2.0000 0.00 3.5000 0.50 5.0000 1.00 

2.2000 0.25 4.0000 0.75 

Point Value for CREW PRODUCTIVITY ■ 3.0000 

emulative Distribution for SAFETY 

State CDF State CDF State CDF 

1.0000 0.00 2.0000 0.50 5.0000 1.00 

1.6000 0.25 3.3000 0.75 

Point Value for DEVELOPMENT RISK * 8.0000 

Cumulative Distribution for RESOURCE RE0TS IMP 

State CDF State CDF State CDF 

4.0000 0.00 5.0000 0.50 5.5000 1.00 

4.5000 0.25 5.4300 0.75 

emulative Distribution for GROWTH POTENTIAL 

State CDF State CDF State CDF 

6.0000 0.00 7.0000 0.50 8.0000 1.00 

6.5000 0.25 7.5000 0.75 

Cumulative Distribution for SPINOFF POTENTIAL 

State CDF State CDF State CDF 

0.0000 0.00 1.6660 0.50 5.0000 1.00 

0.0000 0.25 3.3330 0.75 


Attribute States for ALTERNATIVE: A&RPROTOTY 

Point Value for INITIAL COST IHPCT * 2.0000 

Cumulative Distribution for OPERATIONS COSTS 

State CDF State CDF State CDF 

3.0000 0.00 3.5000 0.50 5.0000 1.00 

3.4000 0.25 4.8000 0.75 

emulative Distribution for CREW PRODUCTIVITY 

State CDF State CDF State CDF 

9.0000 0.00 9.5000 0.50 10.0000 1.00 

9.2500 0.25 9.7500 0.75 
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Cumulative Distribution for SAFETY 

State CDF State CDF State 

0.0000 0.00 2.0000 0.50 4.WKK) 

1.2000 0.25 2.8000 0.75 

Point Value for DEVELOPMENT RISK * 1.0000 

Cumulative Distribution for RESOURCE REQTS IMP 

State CDF State CDF State 

4.0000 0.00 5.0000 0.50 8.0000 

4.2000 0.25 5.3000 0,75 

Cumulative Distribution for GROWTH POTENTIAL 

State CDF State CDF State 

0.0000 0.00 2.5000 0.50 5.0000 

1.2500 0.25 3.7500 0.75 

Cumulative Distribution for SPINOFF POTENTIAL 

State CDF State CDF State 

0.0000 0.00 1.3330 0.50 4.0000 

0.0000 0.25 2.6670 0.75 


Attribute States for ALTERNATIVE: DIAGES/EPS 


Point Value for INITIAL COST IMPCT * 6.0000 

Cumulative Distribution for OPERATIONS COSTS 

State CDF State CDF State 

7.0000 0.00 8.7000 0.50 9.0000 

7.2000 0.25 8.7000 0.75 

Cumulative Distribution for CREW PRODUCTIVITY 

State CDF State CDF State 

5.0000 0.00 5.5000 0.50 6.0000 

5.2500 0.25 5.7500 0.75 

Cumulative Distribution for SAFETY 

State CDF State CDF State 

3.0000 0.00 5.0000 0.50 8.0000 

4.2000 0.25 6.1000 0.75 

Point Value for DEVELOPMENT RISK * 6.0000 

Cumulative Distribution for RESOURCE REQTS IMP 

State CDF State CDF State 

5.0000 0.00 6.0000 0.50 6.5000 

5.7000 0.25 6.2500 0.75 


Cumulative Distribution for 
State CDF 
7.0000 0.00 
7.7500 0.25 


GROWTH POTENTIAL 
State CDF 
8.5000 0.50 
9.2500 0.75 


State 

10.0000 


Cumulative Distribution 
State CDF 
5.0000 0.00 
5.5000 0.25 


for SPINOFF POTENTIAL 
State CDF 
6.0000 0.50 
6.5000 0.75 


State 

7.0000 


CDF 

.00 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 


C - 21 



Attribute States for ALTERNATIVE: ABDUCT/EPS 


Point Value for INITIAL 

COST IMPCT * 

4.0000 


Cumulative Distribution 

for 

OPERATIONS COSTS 



State CDF 


State CDF 

State 

CDF 

2.0000 0.00 


3.0000 0.50 

4.0000 

1.00 

2.5000 0.25 


3.5000 0.75 



Point Value for CREW PRODUCTIVITY * 

5.0000 


Point Value for SAFETY 

* 5.0000 



Point Value for DEVELOPMENT 

RISK 

3.0000 


Cumulative Distribution 

for 

RESOURCE REQTS IMP 



State CDF 


State CDF 

State 

CDF 

3.0000 0.00 


3.5000 0.50 

4.0000 

1.00 

3.2500 0.25 


3.7500 0.75 



Cumulative Distribution 

for 

GROWTH POTENTIAL 



State CDF 


State CDF 

State 

CDF 

5.0000 0.00 


7.5000 0.50 

10.0000 

1.00 

6.2500 0.25 


8.7500 0.75 



Cumulative Distribution 

for 

SPINOFF POTENTIAL 



State CDF 


State CDF 

State 

CDF 

4.0000 0.00 


6.0000 0.50 

8.0000 

1.00 

5.0000 0.25 


7.0000 0.75 




Attribute States for ALTERNATIVE: STD/TLS/ES 

Point Value for INITIAL 

COST IMPCT 

9.0000 


Cumulative Distribution 

for 

OPERATIONS COSTS 



State CDF 


State CDF 

State 

CDF 

3.0000 0.00 


5.0000 0.50 

8.0000 

1.00 

4.8000 0.25 


5.3000 0.75 



Cumulative Distribution 

for 

CREW PRODUCTIVITY 



State CDF 


State CDF 

State 

CDF 

5.0000 0.00 


5.5000 0.50 

6.0000 

1.00 

5.2500 0.25 


5.7500 0.75 



Cumulative Distribution 

for 

SAFETY 



State CDF 


State CDF 

State 

CDF 

2.0000 0.00 


6.0000 0.50 

8.0000 

1.00 

4.5000 0.25 


6.7000 0.75 



Point Value for DEVELOPMENT 

RISK * 

8.0000 


emulative Distribution 

for 

RESOURCE REQTS IMP 



State CDF 


State CDF 

State 

CDF 

4.0000 0.00 


5.0000 0.50 

6.0000 

1.00 

4.5000 0.25 


5.5000 0.75 



emulative Distribution 

for 

GROWTH POTENTIAL 



State CDF 


State CDF 

State 

CDF 

5.0000 0.00 


7.0000 0.50 

9.0000 

1.00 

6.0000 0.25 


8.0000 0.75 
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Cumulative Distribution 
State CDF 
0.0000 0.00 
0.7500 0.25 


for SPINOFF POTENTIAL 
State CDF 
1.5000 0.50 
2.2500 0.75 


State CDF 
3.0000 1.00 


Attribute States for ALTERNATIVE: AUTR08ASSY 


Point Value for INITIAL COST IMPCT 
Cumulative Distribution for OPERATIONS COSTS 
State CDF State CDF 

3.0000 0.00 3.7000 0.50 

3.6000 0.25 3.7000 0.75 


7.0000 

State CDF 
4.0000 1.00 


Point Value for CREW PRODUCTIVITY 
Cumulative Distribution for SAFETY 

State CDF State CDF 

2.0000 0.00 7.0000 0.50 
5.7000 0.25 7.4000 0.75 


4.0000 

State CDF 
8.0000 1.00 


Point Value for DEVELOPMENT RISK 


Cumulative Distribution 
State CDF 

4.0000 0.00 
4.2500 0.25 


for RESOURCE REQTS IMP 
State CDF 
4.5000 0,50 
4.7500 0.75 


9.0000 

State CDF 
5.0000 1.00 


Cumulative Distribution for 
State CDF 
6.0000 0.00 

6.0000 0.25 


GROWTH POTENTIAL 
State CDF 

6.0000 0.50 

7.0000 0.75 


Cumulative Distribution 
State CDF 
0.0000 0.00 
0.0000 0.25 


for SPINOFF POTENTIAL 
State CDF 
0.0000 0.50 

2.0000 0.75 


State CDF 
8.0000 1.00 


State CDF 
4.0000 1.00 


Attribute States for ALTERNATIVE: TRTHERMPOS 


Cumulative Distribution 

for 

INITIAL COST IMPCT 



State CDF 


State CDF 

State 

COF 

5.0000 0.00 


6.0000 0.50 

7.0000 

1.00 

5.5000 0.25 


6.5000 0.75 



Cumulative Distribution 

for 

OPERATIONS COSTS 



State CDF 


State CDF 

State 

CDF 

5.0000 0.00 


6.0000 0.50 

7.0000 

1.00 

5.5000 0.25 


6.5000 0.75 



Point Value for CREW PRODUCTIVITY * 

3.0000 


Cumulative Distribution 

for 

SAFETY 



State CDF 


State CDF 

State 

CDF 

1.0000 0.00 


2.0000 0.50 

3.0000 

1.00 

1.6000 0.25 


2.4000 0.75 
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Point Value for DEVELOPMENT RISK 


Cumulative Distribution 
State CDF 

4.0000 0.00 
4.2500 0.25 


for RESOURCE REGTS IMP 
State COF 
4.5000 0.50 
4.7500 0.75 


8.0000 

State CDF 
5.0000 1.00 


Cumulative Distribution for 
State CDF 

5.0000 0.00 
5.5000 0.25 


GROWTH POTENTIAL 
State CDF 
6.0000 0.50 
6.5000 0.75 


State CDF 
7.0000 1.00 


Cumulative Distribution 
State CDF 
0.0000 0.00 
0.0000 0.25 


for SPINOFF POTENTIAL 
State CDF 

1.0000 0.50 

2.0000 0.75 


State CDF 
3.0000 1.00 


Attribute States for ALTERNATIVE: EVARETRIEV 


Point Value for INITIAL 

COST IMPCT 

4.0000 


Cumulative Distribution 

for 

OPERATIONS COSTS 



State CDF 


State CDF 

State 

CDF 

3.0000 0.00 


5.0000 0.50 

6.0000 

1.00 

4.5000 0.25 


5.4000 0.75 



Cumulative Distribution 

for 

CREW PRODUCTIVITY 



State CDF 


State CDF 

State 

CDF 

4.0000 0.00 


4.5000 0.50 

5.0000 

1.00 

4.2500 0.25 


4.7500 0.75 



CumHative Distribution 

for 

SAFETY 



State CDF 


State CDF 

State 

COF 

1.0000 0.00 


2.0000 0.50 

6.0000 

1.00 

1.3000 0.25 


3.5000 0.75 



Point Value for DEVELOPMENT 

RISK 

5.0000 


Cumulative Distribution 

for 

RESOURCE REQTS IMP 



State CDF 


State CDF 

State 

CDF 

4.0000 0.00 


5.5000 0.50 

7.0000 

1.00 

5.4000 0.25 


5.7000 0,75 



Cumulative Distribution 

for 

GROWTH POTENTIAL 



State CDF 


State COF 

State 

CDF 

5.0000 0.00 


7.0000 0.50 

9.0000 

1.00 

6.0000 0.25 


8.0000 0.75 



Cumulative Distribution 

for 

SPINOFF POTENTIAL 



State CDF 


State CDF 

State 

CDF 

5.0000 0.00 


7.0000 0.50 

9.0000 

1.00 

6.0000 0.25 


8.0000 0.75 




Attribute States for ALTERNATIVE: AUTROBSIH 

Cumulative Distribution for 

INITIAL COST IMPCT 



State CDF 

State CDF 

State 

CDF 

4.0000 0.00 

4.5000 0.50 

5.0000 

1.00 

4.2500 0.25 

4.7500 0.75 
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Cumulative Distribution 

for OPERATIONS COSTS 



State CDF 


State CDF 

State 

CDF 

3.0000 0.00 


4.0000 0.50 

4.8000 

1.00 

3.9000 0.25 


4.4000 0.75 



Point Value for CREW PRODUCTIVITY 

5.0000 


Cunulative Distribution 

for 

SAFETY 



State CDF 


State CDF 

State 

CDF 

0.0000 0.00 


2.0000 0.50 

4.0000 

1.00 

1.2000 0.25 


2.7000 0.75 



Point Value for DEVELOPMENT 

RISK 

4.0000 


Cunulative Distribution 

for 

RESOURCE REQTS IMP 



State CDF 


State CDF 

State 

CDF 

3.5000 0.00 


4.0000 0.50 

4.5000 

1.00 

3.8000 0.25 


4.3000 0.75 



Cumulative Distribution 

for 

GROWTH POTENTIAL 



State CDF 


State CDF 

State 

CDF 

7.0000 0.00 


8.0000 0.50 

9.0000 

1.00 

7.5000 0.25 


8.5000 0.75 



Cunulative Distribution 

for 

SPINOFF POTENTIAL 



State CDF 


State CDF 

State 

CDF 

3.0000 0.00 


4.0000 0.50 

5.0000 

1.00 

3.5000 0.25 


4.5000 0.75 




Attribute States for ALTERNATIVE: STEREOTELE 


Point Value for INITIAL COST IHPCT 
Cumulative Distribution for OPERATIONS COSTS 
State CDF State CDF 

4.0000 0.00 5.0000 0.50 

4.8000 0.25 5.4000 0.75 


3.0000 

State CDF 
7.0000 1.00 


Point Value for CREW PRODUCTIVITY 
Cumulative Distribution for SAFETY 

State CDF State CDF 

2.0000 0.00 5.0000 0.50 
3.8000 0.25 6.3000 0.75 


0.0000 

State CDF 
8.0000 1.00 


Point Value for DEVELOPMENT RISK 


Cumulative Distribution 
State CDF 
4.5000 0.00 
4.9000 0.25 


for RESOURCE REQTS IMP 
State CDF 

5.0000 0.50 
5.6000 0.75 


5.0000 

State CDF 
6.0000 1.00 


Cunulative Distribution 
State CDF 
6.0000 0.00 
6.7500 0.25 


for GROWTH POTENTIAL 
State CDF 
7,5000 0.50 
8.2500 0.75 


State CDF 
9.0000 1.00 


Cumulative Distribution 
State CDF 

5.0000 0.00 

6.0000 0.25 


for SPINOFF POTENTIAL 
State CDF 

7.0000 0.50 

8.0000 0.75 


State CDF 
9.0000 1.00 
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Attribute States for ALTERNATIVE: WORLDMOOEL 


Point Value for INITIAL COST IMPCT 
Cumulative Distribution for OPERATIONS COSTS 


State CDF 
2.0000 0.00 
2.8000 0.25 


State CDF 

3.0000 0.50 
4.2000 0.75 


6.0000 

State CDF 

4.5000 1.00 


Point Value for CREW PRODUCTIVITY 
emulative Distribution for SAFETY 

State CDF State C0F 
0.0000 0.00 2.0000 0.50 
1.2000 0.25 2.7000 0.75 


5.0000 

State CDF 
5.0000 1.00 


Point Value for DEVELOPMENT RISK 


Cumulative Distribution 
State CDF 
0.5000 0.00 
1.8000 0.25 


for RESOURCE REQTS IMP 
State CDF 

2.0000 0.50 
2.1000 0.75 


0.0000 

State CDF 
3.0000 1.00 


Cumulative Distribution for 
State CDF 
0.0000 0.00 

2.5000 0.25 


GROWTH POTENTIAL 
State CDF 
5.0000 0.50 
7.5000 0.75 


State CDF 
10.0000 1.00 


Cumulative Distribution 
State CDF 
0.0000 0.00 

2.5000 0.25 


for SPINOFF POTENTIAL 
State CDF 

5.0000 0.50 

7.5000 0.75 


State CDF 
10.0000 1.00 


Attribute States for ALTERNATIVE: ROBCTLASSY 


Point Value for INITIAL 

COST IMPCT * 

7.0000 


Cumulative Distribution 

for 

OPERATIONS COSTS 



State CDF 


State CDF 

State 

CDF 

3.5000 0.00 


4.0000 0.50 

5.0000 

1.00 

3.9000 0.25 


4.4000 0.75 



Point Value for CREW PRODUCTIVITY 

7.0000 


Cumulative Distribution 

for 

SAFETY 



State CDF 


State CDF 

State 

CDF 

2.0000 0.00 


4.0000 0.50 

7.0000 

1.00 

3.1000 0.25 


4.9000 0.75 



Point Value for DEVELOPMENT 

RISK 

8.0000 


emulative Distribution 

for 

RESOURCE REOTS IMP 



State CDF 


State CDF 

State 

CDF 

4.5000 0.00 


5.0000 0.50 

6.0000 

1.00 

4.9000 0.25 


5.1000 0.75 



Cumulative Distribution 

for 

GROWTH POTENTIAL 



State CDF 


State CDF 

State 

CDF 

6.0000 0.00 


6.0000 0.50 

8.0000 

1.00 

6.0000 0.25 


7.0000 0.75 
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Cunulative Distribution 
State CDF 
0.0000 0.00 
0.0000 0.25 


for SPINOFF POTENTIAL 
State CDF 
1.3330 0.50 
2.6700 0.75 


State CDF 
4.0000 1.00 


Attribute States for ALTERNATIVE: KNOWACQSTD 


Point Value for INITIAL 

COS! 

' IMPCT 

8.0000 


Cumulative Distribution 

for 

OPERATIONS COSTS 



State CDF 


State CDF 

State 

CDF 

3.0000 0.00 


4.0000 0.50 

5.0000 

1.00 

3.2000 0.25 


4.5000 0.75 



Point Value for CREW PRODUCTIVITY 

5.0000 


Cunulative Distribution 

for 

SAFETY 



State CDF 


State CDF 

State 

CDF 

2.0000 0.00 


6.0000 0.50 

8.0000 

1.00 

4.5000 0.25 


6.7000 0.75 



Point Value for DEVELOPMENT 

RISK 

0.0000 


Cunulative Distribution 

for 

RESOURCE REQTS IMP 



State CDF 


State CDF 

State 

CDF 

4.0000 0.00 


5.0000 0.50 

6.0000 

1.00 

4.5000 0.25 


5.5000 0.75 



Cumulative Distribution 

for 

GROWTH POTENTIAL 



State CDF 


State CDF 

State 

CDF 

5.0000 0.00 


6.0000 0.50 

7.0000 

1.00 

5.5000 0.25 


6.5000 0.75 



Cunulative Distribution 

for 

SPINOFF POTENTIAL 



State CDF 


State CDF 

State 

CDF 

0.0000 0.00 


1.5000 0.50 

3.0000 

1.00 

0.7500 0.25 


2.2500 0.75 




Attribute States for ALTERNATIVE: ADAEFFECTI 

Point Value for INITIAL 

COST 

IMPCT » 

8.0000 


Cunulative Distribution 

for OPERATIONS COSTS 



State CDF 


State CDF 

State 

CDF 

4.0000 0.00 


5.0000 0,50 

7.0000 

1.00 

4.7000 0.25 


6.5000 0.75 



Point Value for CREW PRODUCTIVITY 

5.0000 


Cunulative Distribution 

for 

SAFETY 



State CDF 


State CDF 

State 

CDF 

2.0000 0.00 


5.0000 0.50 

8.0000 

1.00 

3.8000 0.25 


6.3000 0.75 



Cunulative Distribution 

for 

DEVELOPMENT RISK 



State CDF 


State CDF 

State 

CDF 

5.0000 0.00 


6.0000 0.50 

7.0000 

1.00 

5.5000 0.25 


6.5000 0.75 
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Cumulative Distribution for RESOURCE REQTS IMP 

State CDF State CDF State 

4.0000 0.00 5.0000 0.50 6.0000 

4.5000 0.25 5.5000 0.75 

Cumulative Distribution for GROWTH POTENTIAL 

State CDF State CDF State 

3.0000 0.00 5.0000 0,50 7.0000 

4.0000 0.25 6.0000 0.75 

Cumulative Distribution for SPINOFF POTENTIAL 

State CDF State CDF State 

5.0000 0.00 6.0000 0.50 7.0000 

5.5000 0.25 6.5000 0.75 


Attribute States for ALTERNATIVE: ES/STDSRES 


Cumulative Distribution for INITIAL COST IMPCT 

State CDF State CDF State 

6.0000 0.00 6.5000 0.50 7.0000 

6.2500 0.25 6,7500 0.75 

Cumulative Distribution for OPERATIONS COSTS 

State CDF State CDF State 

1.00000.00 3.00000.50 5.0000 

1.5000 0.25 3.2000 0.75 

Point Value for CREW PRODUCTIVITY * 5.0000 

Cumulative Distribution for SAFETY 

State CDF State CDF State 

1.0000 0.00 6.0000 0.50 9.0000 

3,8000 0.25 7.1000 0.75 

Point Value for DEVELOPMENT RISK * 4.0000 

Cumulative Distribution for RESOURCE REQTS IMP 

State CDF State CDF State 

4.0000 0.00 5.0000 0.50 6.0000 

4.5000 0.25 5.5000 0.75 

Cumulative Distribution for GROWTH POTENTIAL 

State CDF State CDF State 

5.0000 0.00 6.5000 0.50 8.0000 

5.7500 0.25 7.2500 0.75 

emulative Distribution for SPINOFF POTENTIAL 

State CDF State CDF State 

0.0000 0.00 1.5000 0.50 3.8000 

0.7500 0.25 2.2500 0.75 

Attribute States for ALTERNATIVE: KBSDVMAINT 


Point Value for INITIAL COST IMPCT » 6.0000 
Cumulative Distribution for OPERATIONS COSTS 

State CDF State CDF State 
3.0000 0.00 4.0000 0.50 5.0000 
3.7000 0.25 4.5000 0.75 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 
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Point Value for CREW PRODUCTIVITY - 5.0000 

Cumulative Distribution for SAFETY 

State CDF State CDF State 

2.0000 0.00 6.0000 0.50 8.0000 

4.5000 0.25 6.8000 0.75 

Point Value for DEVELOPMENT RISK = 10.0000 

Cumulative Distribution for RESOURCE REQTS IMP 

State CDF State CDF State 

4.0000 0.00 5.0000 0.50 6.0000 

4.5000 0.25 5.5000 0.75 

Cumulative Distribution for GROWTH POTENTIAL 

State CDF State CDF State 

0.0000 0.00 5.0000 0.50 10.0000 

2.5000 0.25 7.5000 0.75 

Cumulative Distribution for SPINOFF POTENTIAL 

State CDF State CDF State 

0.00000.00 1.50000.50 3.0000 

0.7500 0.25 . 2.2500 0.75 


Attribute States for ALTERNATIVE: DUMMY 


Cumulative Distribution for INITIAL COST IMPCT 

State CDF State CDF State 

0.0000 0.00 0.0000 0.50 10.0000 

0.0000 0.25 0.0000 0.75 

Cumulative Distribution for OPERATIONS COSTS 

State CDF State CDF State 

0.0000 0.00 0.0000 0.50 10.0000 

0.0000 0.25 0.0000 0.75 

Cumulative Distribution for CREW PRODUCTIVITY 

State CDF State CDF State 

0.0000 0.00 0.0000 0.50 10.0000 

0.0000 0.25 0.0000 0.75 

Cunulative Distribution for SAFETY 

State CDF State CDF State 

0.0000 0.00 0.0000 0.50 10.0000 

0.0000 0.25 0.0000 0.75 

Cumulative Distribution for DEVELOPMENT RISK 

State CDF State CDF State 

0.0000 0.00 0.0000 0.50 10.0000 

0.0000 0.25 0.0000 0.75 

Cumulative Distribution for RESOURCE REQTS IMP 

State CDF State CDF State 

0.0000 0.00 0.0000 0.50 10.0000 

0.0000 0.25 0.0000 0.75 


CDF 

1.00 


CDF 

.00 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 


CDF 

1.00 
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State CDF 
10.0000 1.00 


Cumulative Distribution for GROWTH POTENTIAL 
State CDF State CDF 

0.0000 0.00 0.0000 0.50 

0.0000 0.25 0.0000 0.75 

Cumulative Distribution for SPINOFF POTENTIAL 

State CDF State CDF State CDF 

0.0000 0.00 0.0000 0.50 10.0000 1.00 

0.0000 0.25 0.0000 0.75 


B, The Preference Inputs for Each Decision Maker 
Attribute Preferences for Individual: INTRVW 01 


Scaling Constants (Weights) for Each Attribute Are: 
INITIAL COST IMPCT : 0.5000 

OPERATIONS COSTS: 0.7000 

CREW PRODUCTIVITY: 0.6500 

SAFETY: 0.5500 

DEVELOPMENT RISK: 0.4500 

RESOURCE REQTS IMP: 0.3500 

GROWTH POTENTIAL: 0.6000 

SPINOFF POTENTIAL: 0.2500 


The Utility Functions for Each Attribute Are: 
ATTRIBUTE: INITIAL COST IMPCT 


State 

Ut i l i ty 

State 

Utility 

State 

Utility 

0.0000 

0.00 

5.0000 

0.50 

10.0000 

1.00 

ATTRIBUTE: OPERATIONS COSTS 





State 

Utility 

State 

Utility 

State 

Utility 

0.0000 

0.00 

4.0000 

0.50 

10.0000 

1.00 

ATTRIBUTE: CREW PRODUCTIVITY 





State 

Utility 

State 

Utility 

State 

Utility 

0.0000 

0.00 

4.0000 

0.50 

10.0000 

1.00 

ATTRIBUTE; SAFETY 






State 

Utility 

State 

Utility 

State 

Utility 

0.0000 

0.00 

4.0000 

0.50 

10.0000 

1.00 

ATTRIBUTE: DEVELOPMENT RISK 





State 

Utility 

State 

Ut i l i ty 

State 

Utility 

0.0000 

0.00 

5.0000 

0.50 

10.0000 

1.00 

ATTRIBUTE: RESOURCE REQTS IMP 





State 

Utility 

State 

Utility 

State 

Ut i l i ty 

0.0000 

0.00 

5.0000 

0.50 

10.0000 

1.00 

ATTRIBUTE: GROWTH 

POTENTIAL 





State 

Utility 

State 

Utility 

State 

Utility 

0.0000 

0.00 

6.0000 

0.50 

10.0000 

1.00 

ATTRIBUTE: SPINOFF POTENTIAL 





State 

Utility 

State 

Utility 

State 

Utility 

0.0000 

0.00 

4.0000 

0.50 

10.0000 

1.00 
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Attribute Preferences for Individual: INTRVW 02 

Scaling Constants (Weights) for Each Attribute Are: 

INITIAL COST IMPCT: 0.5000 

OPERATIONS COSTS: 0.3500 

CREW PRODUCTIVITY: 0.3000 

SAFETY: 0.5000 

DEVELOPMENT RISK: 0.0500 

RESOURCE REQTS IMP: 0.3000 

GROWTH POTENTIAL: 0.0500 

SPINOFF POTENTIAL: 0.0500 

The Utility Functions for Each Attribute Are; 

ATTRIBUTE: INITIAL COST IMPCT 

State Utility State Utility State Utility 

0.0000 0.00 5.5000 0.50 10.0000 1.00 

ATTRIBUTE: OPERATIONS COSTS 

State Utility State Utility State Utility 

0.0000 0.00 3.5000 0.50 10.0000 1.00 

ATTRIBUTE: CREW PRODUCTIVITY 

State Utility State Utility State Utility 

0.0000 0.00 2.5000 0.50 10.0000 1.00 

ATTRIBUTE: SAFETY 

State Utility State Utility State Utility 

0.0000 0.00 5.5000 0.50 10.0000 1.00 

ATTRIBUTE: DEVELOPMENT RISK 

State Utility State Utility State Utility 

0.0000 0.00 8.0000 0.50 10.0000 1.00 

ATTRIBUTE: RESOURCE REQTS IMP 

State Utility State Utility State Utility 

0.0000 0.00 2.5000 0.50 10.0000 1.00 

ATTRIBUTE: GROWTH POTENTIAL 

State Utility State Utility State Utility 

0.0000 0.00 6.5000 0.50 10.0000 1.00 

ATTRIBUTE: SPINOFF POTENTIAL 

State Utility State Utility State Utility 

0.0000 0.00 9.5000 0.50 10.0000 1.00 


Attribute Preferences for Individual: INTRVW 03 

Scaling Constants (Weights) for Each Attribute Are: 
INITIAL COST IMPCT: 0.2500 

OPERATIONS COSTS: 0.3000 

CREW PRODUCTIVITY: 0.5000 

SAFETY: 0.4000 

DEVELOPMENT RISK: 0.2500 

RESOURCE REQTS IMP: 0.2500 

GROWTH POTENTIAL: 0.2500 

SPINOFF POTENTIAL: 0.2500 
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The Utility Functions for Each Attribute Are: 

ATTRIBUTE: INITIAL COST IMPCT 

State Utility State Utility State Utility 

0.0000 0.00 5.0000 0.50 10.0000 1.00 

ATTRIBUTE: OPERATIONS COSTS 

State Utility State Utility State Utility 

0.0000 0.00 2.5000 0.50 10.0000 1.00 

ATTRIBUTE: CREW PRODUCTIVITY 

State Utility State Utility State Utility 

0.0000 0.00 8.0000 0.50 10.0000 1.00 

ATTRIBUTE: SAFETY 

State Utility State Utility State Utility 

0.0000 0.00 8.0000 0.50 10.0000 1.00 

ATTRIBUTE: DEVELOPMENT RISK 

State Utility State Utility State Utility 

0.0000 0.00 5.0000 0.50 10.0000 1.00 

ATTRIBUTE: RESOURCE REQTS IMP 

State Utility State Utility State Utility 

0.0000 0.00 5.0000 0.50 10.0000 1.00 

ATTRIBUTE: GROWTH POTENTIAL 

State Utility State Utility State Utility 

0.0000 0.00 5.0000 0.50 10.0000 1.00 

ATTRIBUTE: SPINOFF POTENTIAL 

State Utility State Utility State Utility 

0.0000 0.00 5.0000 0.50 10.0000 1.00 


Attribute Preferences for Individual: INTRVU 04 

Scaling Constants (Weights) for Each Attribute Are: 

INITIAL COST IMPCT: 0.4000 

OPERATIONS COSTS: 0.7000 

CREW PRODUCTIVITY: 0.7000 

SAFETY: 0.5000 

DEVELOPMENT RISK: 0.5500 

RESOURCE REQTS IMP: 0.5500 

GROWTH POTENT I AL : 0 . 4000 

SPINOFF POTENTIAL: 0.2500 

The Utility Functions for Each Attribute Are: 

ATTRIBUTE: INITIAL COST IMPCT 

State Utility State Utility State Utility 

0.0000 0.00 4.5000 0.50 10.0000 1.00 

ATTRIBUTE: OPERATIONS COSTS 

State Utility State Utility State Utility 

0.0000 0.00 4.5000 0.50 10.0000 1.00 
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ATTRIBUTE: CREW PRODUCTIVITY 





State 

Utility 

State 

Utility 

State 

Utility 

0.0000 

0.00 

4.0000 

0.50 

10.0000 

1.00 

ATTRIBUTE: SAFETY 






State 

Utility 

State 

Utility 

State 

Utility 

0.0000 

0.00 

3.5000 

0.50 

10.0000 

1.00 

ATTRIBUTE: DEVELOPMENT RISK 





State 

Utility 

State 

Utility 

State 

Uti l i ty 

0.0000 

0.00 

5.5000 

0.50 

10.0000 

1.00 

ATTRIBUTE: RESOURCE REQTS IMP 





State 

Utility 

State 

Utility 

State 

Utility 

0.0000 

0.00 

4.5000 

0.50 

10.0000 

1.00 

ATTRIBUTE: GROWTH 

POTENTIAL 





State 

Utility 

State 

Ut i l i ty 

State 

Ut i l i ty 

0.0000 

0.00 

6.0000 

0.50 

10.0000 

1.00 

ATTRIBUTE: SPINOFF POTENTIAL 





State 

Utility 

State 

Utility 

State 

Utility 

0.0000 

0.00 

2.0000 

0.50 

10.0000 

1.00 


Attribute Preferences for Individual: INTRVW 05 


Scaling Constants (Weights) for Each Attribute Are: 


INITIAL COST INPCT: 0.6500 

OPERATIONS COSTS: 0.6500 

CREW PRODUCTIVITY: 0.6500 

SAFETY: 0.2000 

DEVELOPMENT RISK: 0.5000 

RESOURCE REQTS IMP: 0.5000 

GROWTH POTENTIAL: 0.6000 

SPINOFF POTENTIAL: 0.4000 


The Utility Functions for Each Attribute Are: 
ATTRIBUTE: INITIAL COST IMPCT 

State Utility State Utility 

0.0000 0.00 6.2980 0.50 


State Utility 

10.0000 1.00 


ATTRIBUTE: OPERATIONS COSTS 
State Utility 
0.0000 0.00 


State Utility 
3.0000 0.50 


State Utility 

10.0000 1.00 


ATTRIBUTE: CREW PRODUCTIVITY 
State Utility 

0.0000 0.00 


State Utility 
4.5000 0.50 


State Utility 

10.0000 1.00 


ATTRIBUTE: SAFETY 

State Utility 
0.0000 0.00 


State Utility 
4.0000 0.50 


State Utility 

10.0000 1.00 


ATTRIBUTE: DEVELOPMENT RISK 
State Utility 
0.0000 0.00 


State Utility 
4.7980 0.50 


State Utility 
10.0000 1.00 
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ATTRIBUTE: RESOURCE REOTS IMP 

State Utility State Utility State Utility 

0.0000 0.00 3.0000 0.50 10.0000 1.00 

ATTRIBUTE: GROWTH POTENTIAL 

State Utility State Utility State Utility 

0.0000 0.00 6.0000 0.50 10.0000 1.00 


ATTRIBUTE: SPINOFF POTENTIAL 

State Utility State Utility State Utility 

0.0000 0.00 5.5000 0.50 10.0000 1.00 


Attribute Preferences for Individual: INTRVW 06 

Scaling Constants (Weights) for Each Attribute Are: 

INITIAL COST IMPCT : 0.5000 

OPERATIONS COSTS: 0.7500 

CREW PRODUCTIVITY: 0.7200 

SAFETY: 0.7500 

DEVELOPMENT RISK: 0.5500 

RESOURCE REOTS IMP: 0.4500 

GROWTH POTENTIAL: 0.5500 

SPINOFF POTENTIAL: 0.4000 

The Utility Functions for Each Attribute Are: 

ATTRIBUTE: INITIAL COST IMPCT — 

State Utility State Utility State Utility 

0.0000 0.00 6.5000 0.50 10.0000 J .00 

ATTRIBUTE: OPERATIONS COSTS 

State Utility State Utility State Utility 

0.0000 0.00 7.5000 0.50 10.0000 1.00 

ATTRIBUTE: CREW PRODUCTIVITY 

State Utility State Utility State Utility 

0.0000 0.00 5.0000 0.50 10.0000 1.00 

ATTRIBUTE: SAFETY 

State Utility State Utility State Utility 

0.0000 0.00 5.0000 0.50 10.0000 1.00 

ATTRIBUTE: DEVELOPMENT RISK 

State Utility State Utility State Utility 

0.0000 0.00 4.0000 0.50 10.0000 1.00 

ATTRIBUTE: RESOURCE REQTS IMP 

State Utility State Utility State Utility 

0.0000 0.00 6.5000 0.50 10.0000 1.00 

ATTRIBUTE: GROWTH POTENTIAL 

State Utility State Utility State Utility 

0.0000 0.00 6.0000 0.50 10.0000 1.00 
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ATTRIBUTE: SPINOFF POTENTIAL 

State Utility State Utility State Utility 

0.0000 0.00 5.5000 0.50 10.0000 1.00 


Attribute Preferences for Individual: INTRVW 07 

Scaling Constants (Weights) for Each Attribute Are: 

INITIAL COST IMPCT : 0.3000 

OPERATIONS COSTS: 0.2000 

CREW PRODUCTIVITY: 0.3000 

SAFETY: 0.2000 

DEVELOPMENT RISK: 0.3000 

RESOURCE REQTS IMP: 0.2000 

GROWTH POTENTIAL: 0,3000 

SPINOFF POTENTIAL: 0.1000 

The Utility Functions for Each Attribute Are: 

ATTRIBUTE: INITIAL COST IMPCT 

State Utility State Utility State Utility 

0.0000 0.00 4.0000 0.50 10.0000 1.00 

ATTRIBUTE: OPERATIONS COSTS 

State Utility State Utility State Utility 

0.0000 0.00 3.0000 0.50 10.0000 1.00 

ATTRIBUTE: CREW PRODUCTIVITY 

State Utility State Utility State Utility 

0.0000 0.00 7.0000 0.50 10.0000 1.00 

ATTRIBUTE: SAFETY 

State Utility State Utility State Utility 

0.0000 0.00 6.0000 0.50 10.0000 1.00 

ATTRIBUTE: DEVELOPMENT RISK 

State Utility State Utility State Utility 

0.0000 0.00 5.0000 0.50 10.0000 1.00 

ATTRIBUTE: RESOURCE REQTS IMP 

State Utility State Utility State Utility 

0.0000 0.00 4.0000 0.50 10.0000 1.00 

ATTRIBUTE: GROWTH POTENTIAL 

State Utility State Utility State Utility 

0.0000 0.00 7.0000 0.50 10.0000 1.00 

ATTRIBUTE: SPINOFF POTENTIAL 

State Utility State Utility State Utility 

0.0000 0.00 3.0000 0.50 10.0000 1.00 
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Attribute Preferences for Individual: INTRVW 08 

Scaling Constants (Weights) for Each Attribute Are: 
INITIAL COST IHPCT: 0.5500 

OPERATIONS COSTS: 0.5500 

CREW PRODUCTIVITY: 0.4500 

SAFETY: 0.4500 

DEVELOPMENT RISK: 0.4500 

RESOURCE REQTS IMP: 0.4500 

GROWTH POTENTIAL: 0.4000 

SPINOFF POTENTIAL: 0.3000 


The Utility Functions for Each Attribute Are: 
ATTRIBUTE: INITIAL COST IMPCT 


State 

Utility 

State 

Utility 

State 

Utility 

0.0000 

0.00 

5.0000 

0.50 

10.0000 

1.00 

ATTRIBUTE: OPERATIONS COSTS 





State 

Utility 

State 

Utility 

State 

Utility 

0.0000 

0.00 

3.5000 

0.50 

10.0000 

1.00 

ATTRIBUTE: CREW PRODUCTIVITY 





State 

Utility 

State 

Utility 

State 

Utility 

0.0000 

0.00 

5.0000 

0.50 

10.0000 

1.00 

ATTRIBUTE: SAFETY 






State 

Utility 

State 

Utility 

State 

Ut i l i ty 

0.0000 

0.00 

4.0000 

0.50 

10.0000 

1.00 

ATTRIBUTE: DEVELOPMENT RISK 





State 

Utility 

State 

Utility 

State 

Utility 

0.0000 

0.00 

5.0000 

0.50 

10.0000 

1.00 

ATTRIBUTE: RESOURCE REQTS IMP 





State 

Utility 

State 

Utility 

State 

Utility 

0.0000 

0.00 

3.5000 

0.50 

10.0000 

1.00 

ATTRIBUTE: GROWTH 

POTENTIAL 





State 

Utility 

State 

Utility 

State 

Utility 

0.0000 

0.00 

5.0000 

0.50 

10.0000 

1.00 

ATTRIBUTE: SPINOFF POTENTIAL 





State 

Utility 

State 

Utility 

State 

Utility 

0.0000 

0.00 

5.0000 

0.50 

10.0000 

1.00 


Attribute Preferences for Individual: INTRVW 09 

Scaling Constants (Weights) for Each Attribute Are: 
INITIAL COST IMPCT: 0.7000 

OPERATIONS COSTS: 0.7000 

CREW PRODUCTIVITY : 0.6000 

SAFETY: 0.6000 

DEVELOPMENT RISK: 0.5000 

RESOURCE REQTS IMP: 0.6000 

GROWTH POTENTIAL: 0.5000 

SPINOFF POTENTIAL: 0.5000 
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The Utility Functions for Each Attribute Are: 


ATTRIBUTE: INITIAL COST IMPCT 
State Utility 

0.0000 0.00 

ATTRIBUTE: OPERATIONS COSTS 
State Utility 

0.0000 0.00 

ATTRIBUTE: CREW PRODUCTIVITY 
State Utility 

0.0000 0.00 

ATTRIBUTE: SAFETY 

State Utility 

0.0000 0.00 

ATTRIBUTE: DEVELOPMENT RISK 
State Utility 

0.0000 0.00 

ATTRIBUTE: RESOURCE REQTS IMP 
State Utility 

0.0000 0.00 

ATTRIBUTE: GROWTH POTENTIAL 
State Utility 

0.0000 0.00 

ATTRIBUTE: SPINOFF POTENTIAL 
State Utility 

0.0000 0.00 


State 

Utility 

State 

7.0000 

0.50 

10.0000 

State 

Utility 

State 

6.0000 

0.50 

10.0000 

State 

Utility 

State 

6.0000 

0.50 

10.0000 

State 

Utility 

State 

5.0000 

0.50 

10.0000 

State 

Utility 

State 

5.0000 

0,50 

10.0000 

State 

Utility 

State 

5.0000 

0.50 

10.0000 

State 

Utility 

State 

5.0000 

0.50 

10.0000 

State 

Utility 

State 

5.0000 

0.50 

10.0000 


Utility 

1.00 


Utility 

1.00 


Utility 

1.00 


Utility 

1.00 


Utility 

1.00 


Utility 

1.00 


Utility 

1.00 


Utility 

1.00 
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Appendix D: Post-Process Questionnaire 


Jet Propulsion Laboratory 
California Institute of Technology 

4800 Ook Grove Drive 
Pasadena, California 91109 

(818) 354-4321 

August 10, 1989 



Dear Colleague: 

Recently you participated In an evaluation of the Automation and Robotics (A&R) 
proposals which had been submitted as candidates for high leverage (moderate- 
to-high risk, high payoff) A&R prototyping. Enclosure 2 summarizes the results 
of that activity. Background materials are enclosed together with the results 
of the ranking as Enclosure 3. This package was assembled to Inform you of the 
results and thank you for your participation. 

Ve are Interested in your complete and candid evaluation of the entire process, 
from Initial proposal submissions to final ranking. Enclosure 1 was prepared 
to obtain those evaluations. Your evaluations will assist us in exposing any 
weaknesses in the process. They will also permit us to improve the process by 
modifying our questionnaires, Interviews, etc., to circumvent such problems. 
Either signed or anonymous quest ionnaires can be returned. No individual 
responses will be quoted or identified. Only overall results and summaries will 
be used in reporting back to the Space Station Program Office. 

Three categories of participants were involved (technical assessors, decision 
makers [Interviewees], and Interested observers). Please Identify your 
participation category and respond as completely as possible to the questions. 
The questionnaire is divided Into three sections. Section A exnlores your 
overall reaction to the evaluation process. Section B refers to the technical 
assessment. Section C applies to the prioritization process. 

Completed ques tionnaires should be returned by August 30, 1989. An addressed 
envelope is provided for your convenience. Reston participants can return the 
questionnaire to Ms. E. Carpenter (SSR) . If you did not attend the June briefing 
and have a question, please contact me at tel. (818) 354-1236 (TTS 792-1236). 
Once again, thank you for your participation. 


Dr. Jeffrey H. Smith 
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Enclosures 





A&R PROPOSAL EVALUATION QUESTIONNAIRE 10 August, 1989 

Please complete the following by checking the appropriate box (NA indicates "no answer”; e.g., not familiar with topic) 


I participated in the evafuation as a: 

□ Technical assessor (attribute ratings) □ Decision maker (prioritized the attributes) □ Interested Observer 


A. ' 

YOUR OVERALL REACTION TO THE EVALUATION Unsatisfactory 

Poor 

Fair 

Good 

Excellent 

NA 

(i) 

Describe the value of ranking the A&R candidates 








in improving your understanding of them. 

□ 

□ 

□ 

D 

□ 

□ 

(2) 

Describe the value of rankings for evaluating A&R candidates. 

□ 

□ 

□ 

□ 

□ 

□ 

(3) 

Describe the usefulness of the results for measuring candidate 








value. 

□ 

0 

□ 

□ 

□ 

□ 

(4) 

Describe the effectiveness of the safeguards in preventing 








"gaming'’ by the participants. 

□ 

□ 

□ 

□ 

□ 

□ 

(5) 

Describe the fairness of the process from the proposers’ 








point of view. 

□ 

□ 

□ 

□ 

□ 

□ 

(6) 

Describe the fairness of the process from your 








point of view. 

□ 

□ 

□ 

□ 

□ 

□ 

(7) 

Describe the efficiency of the process for generating final 








rankings 

□ 

□ 

a 

0 

□ 

0 

(8) 

After examining the results, describe your intuitive 








concurrence. 

□ 

n 

□ 

□ 

□ 

□ 

(9) 

Describe the utility to you of similar processes in future 








evaluations. 

□ 

□ 

□ 

□ 

□ 

□ 

1 

B. 

THE TECHNICAL ASSESSMENT PROCESS 







(i) 

Describe the reasonableness of the groundrules. 

□ 

0 

□ 

□ 

□ 

□ 

(2) 

Describe the adequacy of the process in addressing the 








uncertainties in candidate attributes. 

0 

□ 

□ 

□ 

□ 

□ 

(3) 

Describe the validity of using arguments for and against 








the attribute ratings in establishing technical credibility. 

□ 

□ 

□ 

□ 

□ 

□ 

(4) 

Describe the credibility of the technical assessment 








process. 

0 

□ 

n 

□ 

□ 

□ 

(5) 

Describe the adequacy of the technical assessment 








process in preventing "gaming.” 

□ 

□ 

□ 

□ 

□ 

□ 

1 

C. THE ASSESSMENT OF PRIORITIES 







(1) 

Describe the adequacy of SSF Decision Maker 








representation, given the scope of the ^valuation. 

□ 

□ 

□ 

□ 

□ 

□ 

(2) 

Describe the utility of the group decision rules for 








expressing group preferences. 

□ 

□ 

□ 

□ 

□ 

□ 


(3) If asked to participate in a future evaluation, would you prefer a process with: 

□ Analyst support only? □ Interactive computer support □ Stand-alone computer software 

with analyst backup? with analyst coordination? 

ADDITIONAL COMMENTS: 
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